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Since its inception in 1988, the Board on 
International Comparative Studies in Education (BICSE) has monitored 
U.S. participation in those cross national comparative studies in 
education that are funded by its sponsors, the National Science 
Foundation and the National Center for Education Statistics. This set 
of international study descriptions represents a status report on 17 
projects presented to BICSE at various times in 1994. Reports were 
prepared by the projects themselves and are presented without 
evaluation or editing. The following are described: (l) "Case Studies 
of U.S. Innovations in Mathematics, Science, and Technology in an 
International Context" (National Center for Improving Science 
Education and other agencies); (2) ''Civics Education Study" 
(International Association for the Evaluation of Educational 
Achievement); (3) "Comparative Study of Teacher Training and 
Professional Development Practices in APEC (Asia Pacific Economic 
Co-operation) Members" (Asia Pacific Economic Co-operation 
Ministerial); (A) "Computers in Education Study" (International 
Association for the Evaluation of Educational Achievement [lEA]); (5) 
"Cross-national Studies of Adult Understanding of Science" (Chicago 
Academy of Sciences); (6) "Education Indicators Project (INES)" 
(Organization for Economic Cooperation and Development); (7) 
•^International Adult Literacy Study" (Statistics Canada); (8) 
"International Assessment of Educational Progress" (Educational 
Testing Service); (9) "International Comparative Study of Mathematics 
and Science Tests" (National Center for Improving Science Education); 
(10) "International Study of Teachers' Attitudes toward Reform and 
Teacher Preparation for Implementing Reform" (George Washington 
University Institute for Curriculum Standards and Technology); (11) 
"Language Education Study" (IEa); (12) "New Standards Project 
(benchmark activities)" (Learning Research and Development Center at 
the University of Pittsburgh and National Center on Education and the 
Economy); (13) "Preprimary Project" (IEA); (lA) "Reading Literacy 
Study" (IEA); (15) *'Study on Performance Standards in Education" 
(U.S. Department of Education); (16) "Survey of Mathematics and 
Science Opportunity" (Michigan State University); and (17) "Third 
International Mathematics and Science Study (TIMSS) (including 
O special U.S. TIMSS activities)" (IEA). (SLD) 
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The National Academy of Sciences is a private, nonprofit, self-perpetuating society of 
distinguished scholars engaged in scientific and engineering research, dedicated to the furtfierance of 
science and technology and to their use for the general welfare. Upon the authority of the charter 
granted to it by the Congress m 1863, the Academy has a mandate that requires it to advise the 
federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National 
Academy of Sciences. 

The National Academy of Engineering was established in 1964, under the charter of the 
National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous 
in its administration and in the selection of its members, sharing with the National Academy of 
Sciences the responsibility for advising the federal government. The National Academy of 
Engineering also sponsors engineering programs aimed at meeting national needs, encourages 
education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White 
is president of the National Academy of Engineering. 

The Institute of Medicine was established in 1970 by the National Academy of Sciences to 
secure the services of eminent members of impropriate professions in the examination of policy 
matters pertaining to the health of the public. The Institute acts under the responsibility given to the 
National Academy of Sciences by its congressional charter to be an adviser to the federal government 
and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth 
I. Shine is president of the Institute of Medicine. 

The National Research Council was organized by the National Academy of Sciences in 1916 
to associate the broad community of science and technology with the Academy's purposes of 
furthering knowledge and advising the federal government. Functioning in accordance with general 
policies determined by the Academy, the Council has become the principal operating agency of both 
the National Academy of Sciences and the National Academy of Engineering in providing services to 
the government, the public, and the scientific and engineering communities. The Council is 
administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. 
Robert M. White are chairman and vice chairman, respectively of the National Research Council. 

The work of the Board on International Comparative Studies in Education is supported with 
funds from the National Science Foundation and the National Center for Education Statistics, through 
a grant from the National Science Foundation. 

Copyright 1995 by the National Academy of Sciences. All rights reserved. 
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Interest in education abroad and how it compares to schooling in the United States has been rising 
among policy makers, researchers, and practitioners. Since its inception in 1988, the Board on 
International Comparative Studies in Education (BICSE) has monitored U.S. participation in those 
cross*national comparative studies in education that are funded by its sponsors, the National Center 
for Education Statistics and the National Science Foundation. It has also provided a forum for 
information and discussion, which has led it to request presentations on selected additional studies of 
special interest to the board and its sponsors. 

As a result, BICSE has in its records information about the aims, designs, and intended outcomes of a 
number of the largest international studies in education thr^ have been undertaken in recent years. 
BICSE*s collection clearly does not cov^r all or even a representative sampling of the broad range of 
studies available on comparative and international education issues. Nevertheless, BICSE's 
accumulated information represents a valuable resource on a variety of studies, a compilation that is 
not available elsewhere. Participants at the Second International Conference on Long*Range Planning 
for Large-scale Collections of Intanational Education Statistics convened by BICSE in January 1993 
urged BICSE to make such international study information available to interested individuals in the 
United States and in other countries as well. 

This set of international study descriptions rq>resents a status rq>ort on 17 projects presented to 
BICSE. The study descriptions were compiled by Jane Phillips of the BICSE staff from information 
submitted to the board by each project's staff; each compilation was then submitted to its respective 
project director for editing and updating. These updates were accomplished at various times in 1994, 
as indicated in the individual project descriptions. BICSE has not evaluated or edited the 
descriptions, but merely presents them as reports by the projects themselves on their activities. Each 
study description indicates key contact persons for that project who can provide further information 
and subsequent updates on progress and findings. 

The Board on International Comparative Studies in Education has a limited number of copies of this 
collection of descriptions for complementary distribution. (2101 Constitution Avenue, Harris 178, 
Washington, D.C., 20418, U.S.A.; telephone: 1/202/334-3010; facsimile: 1/202/334-3584) 

The document will al^o be made available on Internet through World Wide Web. It may be accessed 
by using the World Wide Web address, (http://www.nas.edu/) 

The U.S. Department of Education Educational Resources Information Center (ERIC) will have the 
document on microfiche. (ERIC, 1301 Piccard Dr., Suite 300, Rockville, Maryland 20850-3238, 
U.S.A.; telephone: 1/301/258-5500) 
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INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION: DESCRIPTIONS 



Case Studies ofU.S. Innovations in Mathematics^ Science^ and Technology in an 
International Context 3 
The National Center for Improving Science Education, National Center for Research in 
Mathematical Sciences Education, and Center for Educational Research and Innovation 
(CERI/Organisation for Economic Co-operation and Development) 

Civics Education Study 7 
International Association for the Evaluation of Educational Achievement 

Comparative Study of Teacher Training and Projessional Development Practices in 

APEC Members 12 

Asia Pacific Economic Co-operation Ministerial 

Computers in Education Study 15 
International Association for the Evaluation of Educational Achievement (lEA) 

Cross-national Studies of Adult Understanding of Science 23 
Chicago Academy of Sciences 

Education Indicators Project (INES) 29 
Organisation for Economic Cooperation and Development 

International Adult Literacy Study 42 
Statistics Canada 

International Assessment of Educational Progress 48 
Educational Testing Service 

International Comparative Study of Mathematics and Science Tests 58 
The National Center for Improving Science Education 

International study of teachers^ attitudes toward reform and teacher preparation 

for implementing reform 62 

George Washington University Institute for Curriculum Standards and Technology 

Language Education Study 65 
International Association for the Evaluation of Educational Achievement (lEA) 

New Standards Project (benchmark activities) 74 
Learning Research and Development Center at the University of Pittsburgh and 
National Center on Education and the Economy 

Preprimary Project 79 

International Association for the Evaluation of Educational Achievement (lEA) 
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Reading Literacy Study 89 
International Association for the Evaluation of Educational Achievement (lEA) 

Study on Performance Standards in Education 97 
U.S. Department of Education, Office of the Under Secretary and the 
Organisation for Economic Cooperation and Development 

Survey of Mathematics and Science Opportunity (SMSO) 101 
Michigan State University 

Third International Mathematics and Science Study (TIMSS) (including special 
U.S. TIMSS acHvities) 106 
International Association for the Evaluation of Educational Achievement (lEA) 
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CASE STUDIES OF U.S. INNOVATIONS IN MATHEMATICS, SCIENCE, AND TECHNOLOGY 
IN AN INTERNATIONAL CONTEXT 



International Organization The National Center for Improving Science Education, National 

Center for Research in Mathematical Sciences Education, Stanford 
University, and Center for Educational Research and Innovation 
(CERI/Organisation for Economic Co-operation and Development) 

Years of Data Cotteetion Phase One 1991-92 

Phase Two - 1992-95 

Purpose The outcome of this two-phase study will be eight intensive case studies on mathematics and 
science innovations, the United States' contribution to an international effort. The project examines 
how innovations in science and mathematics occur around the world. The idea for international case 
studies of innovation in science, mathematics, and technology education grew out of CERI/OECD 
member nations' growing concerns for more effective mathematics, science, and technology education 
programs to serve their populations. They recognized the need to have an in*depth understanding of 
the policies, programs, and practices that lead to successful innovations in science and mathematics 
education* Further, they desired a greater understanding of how these innovations in programs, 
policies, and practices are implemented in different countries and settings. Approaches to reforming 
education developed in one country may be very helpiul to educators elsewhere. 

Organization and Management Coordinated by the Center for Educational Research and Innovation 
(CERI/Organisation for Economic Co-operation and Development [OECD]), the National Center for 
Improving Science Education has the responsibility for the science case studies; the National Center 
for Research in Mathematical Sciences Education has responsibility for the mathematics studies. 
Phase Two will be carried out by the National Center for Improving Science Education, the National 
Center for Research in Mathematical Science of the Wisconsin Center for Education Research, and 
Stanford University. Project staff include The National Center for Improving Science Education 
director and project director, a subcontractor director, and senior research scientists (Michael 
Huberman, Mary Budd Rowe, Myron Atkin, Jeremy Kilpatrick, and Doug McLeod). 

The project includes a subcontract for OECD to catalyze and monitor the data collection in each of 
the participating member countries during their own national studies. Although many more countries 
participated in Phasd One, OECD will attempt to hold the number of countries participating in Phase 
Two to six to eight to facilitate quality control. OECD's monitoring will be carried out through 
science and mathematics education specialists and will be directed toward tailoring the case studies to 
meet the methodological requirements of the project and to ensure that questions of common interest 
are addressed in each country. 

Design 

Participants Phase One - OECD member countries. 

Phase Two - Six to eight OECD member countries. 

Sample In the main^ sampling is based on selecting sites/events that illustrate particularly 
interesting theoretical or applied issues. 
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Procedures a nd Summan of Content 



Phase One. The scope of work for Phase One entailed selecting the U.S. innovations and writing 20- 
page case summaries* To complete this work, the following specific activities took place: 

o An advisory board was established consisting of the U.S. representatives to the several 

planning meetings held by CERI/OECD and added members, including science, mathematics, 
and case study m^odology experts. The advisors will continue to provide guidance 
throughout Phase Two of the study. 

o The staff and advisory board finalized the selection of the innovations that were to be 

described in Phase One. The innovations selected were: California's Systemic Improvement 
of Science Education; Chemistry in the Community (ChemCom); Kids Network; Project 
2061; State of California's Restructuring of Mathematics Education; The National Council of 
Teachers of Mathematics Standards Project; The Urban Mathematics Collaborative Project; 
and The Voyage of the Mimi. 

o The advisory board met to review the papers produced in Phase One and further refine the 
plans for the in-d^th case studies to be carried out in Phase Two. The California 
mathematics project was dropped and Contemporary Pre-Calculus Through Applications 
added. The Phase One ps^ers were revised based on advisory board suggestions and have 
been published by OECD for international distribution. 

Phase Two. The work of Phase Two is to document the philosophy, genesis, implementation and, in 
some cases, routinization of major innovations in science, mathematics, and technology education. A 
case study design provides for conmion issues to be explored and analyzed while still paying attention 
to the unique features of the individual case. 

Data Collection and Amdvses 

Phase Two: The overall approach for the international study is based on the work of Robert Yin 
(1984), Michael Huberman, and Matt Miles (1984), and informed by the work of Robert Stake and 
Jack Easley (1978). Comprehensive, systematic, and in-depth information will be collected through 
field observations guided by the same general questions. However, different methods of inquiry and 
analyses will be used to reveal the most salient features of each of the innovations. 

The case study research will follow events and processes over time, reconstituting milestones that 
occurred before the researchers picked up the case and carefully documenting unfolding events 
through a variety of modes. Some of the research questions will have to be general, covering all 
cases, and others will be project-specific. 

Michael Huberman has developed a model to guide sampling, collecting, and coding data, carrying 
out an intermediate analysis, and conducting the final analysis and write-up* For each project, 
research questions will be grouped, and the coded segments used to respond to each, including 
illustrative material in the form of excerpts, vignettes, and composite sketches. Alternatively, major 
themes, leitnnotifis, dilemmas, and achievements can be explicated across research questions for each 
project. Each case will constitute a narrative, with the research questions converted into chapters and 
the main findings presented at their proper chronological moment. The case studies will not exceed 
80 pages each with a three page synopsis* The field research teams will develop, review, and critique 
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case outlines prior to final write-up; the write-ups alio will be reviewed by a small group of critical 
readers. There also will be a cross-project analysis. 

Phase One: (October 1991 to October 1992) 



1991-1992 An advisory board was established; the staff and advisory board finalized the 
selection of innovations to be described in Phase One. 
The advisory board reviewed Phase One papers and refined plans for Phase 
Two in-dq[)tii case studies. 
Phase One psqpers were revised and published. 



Phase Two: (Sq)tember 1992 to Sq?tember 1995) 

1992-199S Information will be collected through field observations. 
Case study narratives will be published. 
A cross-project analysis will be published. 

Publications Phase One papers have been published by OECD for international distribution. 
Phase Two case study narratives will be published. 
A cross-project analysis will be published. 

Funding Phase One and Phase Two funding is provided by the National Science Foundation 
and the U.S. Department of Education. 



Information Sources 

Senta A. Raizen^ Director 

The National Center for Improving Science Education 
2000 L Street, N.W., Suite 603 
Washington, D.C. 20036 

telephone: 202/ 467-0652 

facsimile: 202/ 467-0659 

Bitnet: sentar@gwuvm.gwu.edu 



Ted Britton, Project Director 

The National Center for Improving Science Education 

2000 L Street, N.W., Suite 603 

Washington, D.C. 20036 

telephone: 202/ 467-0652 

facsimile: 202/ 467-0659. 

e-mail: tbritton@fAruvm.gwu.edu 
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Norman Webb, Senior Researcher 

National Center for Research in Mathematical Sciences Education 
Wisconsin Center for Education Research 
School of Education, University of Wisconsin-Madison 
Madison, Wisconsin 53706 

telephone: 608/ 263«4287 

facsimile: 608/ 263-^8 

e*mail: normwebb@vms.mace.wisc.edu 



National Cent er for Improving Science Education 

1992 Case Studies of U.S. Innovations in Mathematics, Science, and Technology in an International 
Context: Project Sununary. 

Organisation for Economic Cooperation and Development 

1993 Science and Mathematics Education in the United States: Eight Innovations. Publication 
number 96*93-01-1. OECD» Paris, France. (Available from OECD Publications, 
Washington, D.C.) 

Raizen» Senta 

1992 Case Studies of U.S. Innovations in Mathematics* Science, and Technology in an International 
Context. Presentation to the Board on International Con^uirative Studies in Education. 
October. 



NOTE: This study summary was reviewed and edited by Ted Britton at The National Center for 
Improving Science Education in Washington, D.C. on May 31> 1994. 
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CmCS EDUCATION STUDY 



In ternational Organization International Association for the Evaluation of Educational 

Achievement (lEA) 

Years o f Data Collection Case studies: 1995-1996 

Data collection: 1997-98 

Purpose lEA conducted a Civic Education Survey in 1971 as part of a Six-subject Survey of student 
achievement (science, reading comprehension, literature, French as a foreign language, English as a 
foreign language, and civics education). Ten lEA member countries participated in the civics 
education portion. Extensive regression analysis was conducted, with a fairly consistent finding that 
stress on rote learning and on patriotic ritual within the classroom tended to be negatively related to 
civics education outcomes, while the opportunity to express an opinion in class had a positive 
relationship. The socioeconomic status of the family and the type of school (academic or vocational) 
were controlled statistically in the regression analysis, and these findings were extremely similar 
across nine Western industrialized countries with different educational and political contexts. 

Since 1971, many changes have taken place in schools and political systems, which raise new issues 
and intensify concern for old ones. Needs and opportunities for a second civics education study are: 



0 Moves toward democratization around the world 

0 Increasing immigration/emigration and resurgent authoritarianism and racism 

0 Alienation among youth and the growth of what has been called **unconventional 
participation** 

0 Recognition of the implicit or hidden curriculum 

0 Wariness concerning discussions of civics and political 

0 Women's increasing role in politics 

0 Prominent environmental issues 

0 Powerful mass media 



In addition, there have been advances in the social sciences, educational measurement, and the 
methodologies and models used by lEA that provide a background for a second study. 

Many differences exist around the world in the content and process of civics education. In societies 
undergoing rapid social and political change and democratization attempts are being made to prepare 
young people for a political and economic order that does not yet exist. Civics education differs from 
many other school subjects in the relative lack of consensus even within societies with long 
democratic traditions about the knowledge and attitude base, but even more in countries where 
individuals are wary of pleas for national unity and of courses with titles such as civics or political 
education. Cross-national research has a vital role in providing the research base for policy makers, 
those who design curricula, those who prepare teachers, and the general public. 

OmniTHti of^ ond Management An international coordinating center will be designated through 
competitive bidding for Phase II to begin operations in mid- 1996. Until then. Phase I and planning 
for Phase II will take place through ad hoc planning groups as part of a special project involving lEA 
headquarters. 
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Design 

Partitmnts 

Six^subject Survey * civics education: Finland, Germany (Federal Republic), Iran, Ireland, 
Israel, Italy, Netherlands, New Zealand, Sweden, and the United States. (1971) 

Civics Education Study: There is high interest in a civics education study in countries of 
Central and Eastern Europe and Asia, and in countries that belong to the Organisation for 
Economic Co-operation and Development. (1994-2000) 

Samnle. 

Six-subject Survey * civics: The aims of the research were to identify those factors 
accounting for differences between countries, between schools, and between students. The 
technique used was a cross-sectional survey at three different levels, which described 
education as it was at the time of testing and not as it might be. Probability samples of 
schools and students within schools were drawn for each level within each country. 

Civics Education Study: 

Phase I: In some countries, structured interviews will be conducted with small samples of 
students. 

Phase II: The survey of knowledge, attitudes, and participation or behavior will be conducted 
with nationally representative samples. 

Procedures and Summa ry o f Content 

Six-subject Survey * civics: Three international populations were identified: Population I included 
all students in full-time schooling aged 10:0-10:11 years; Population II included all students in full- 
time schooling aged 14:0-14:11 years; and Population IV encompassed all students in the terminal 
year of full-time secondary education programs that were either pre-univeirsity programs or programs 
of the same length. A Population III was designed for national data collection and analysis only. 
Tests were developed to yield a total "cognitive" score* There were survey scales to measure 
attitudinal outcomes as well as perceptions about '*how society works.** 

Civics Education Study: 

Phase I. Each participating country will prepare a case study of its civics education, including 
reviews of previous empirical and policy studies, interviews, and analysis of curricula, to gather: 

0 information about the political, educational, social, and economic context with special 
attention to major recent changes or reforms, including some social indicators 

0 conceptions of and definitions related to civics education, including both official 

statements such as national curricula or assessment standards, and other major points 
of view 

0 information about content and methods of civics education as it is practiced, centered 
on intended and like unintended learning outcomes of civics education and how it is 
envisioned that these outcomes will be achieved by students 
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o a perspective concerning other factors influencing the civics education process and its 
outcomes, such as educational policy more generally, school organization, media, 
parental, and family characteristics 

Among the sources to be used are: 

0 analysis of curriculum and policy documents in areas such as national and world 

history, civics, social studies, moral education, literature, as well as participation 

opportunities provided to students in and out of schools 
o interviews (individually or in focus groups) with policy makers, leaders in 

professional organizations such as teachers unions and experts in teacher training 
o in some countries structured interviews with small samples of students either 

individually, in focus groups, or using computer networks 
o review of existing studies conducted within the country of civics education, moral 

education, youth and adult attitudes, and political socialization 

The national case studies will be conducted around a common framework prepared by an international 
conmiittee. Each participating country will make a proposal to this international committee with 
respect to the types of material they intend to collect for the self-study and how it will be analyzed. 
The international committee will make suggestions about this material and nominate out-of-^country 
experts to examine the resulting data and self-study t&pott in order to give advice to the national 
team. The national team will retain responsibility for the rq>ort itself, and the external panel will 
prepare a report documenting its own views* The case studies and commentaries will be prepared for 
publication, and will also produce guidance for Phase II of the study, with respect to topics to be 
considered, aspects of formal and informal education to be surveyed, outcomes, and predictors. 

Phase IL The second phase will be a survey of knowledge, attitudes, and participation or behavior in 
nationally representative samples. It will be designed around issues and processes and programs 
currently in place and questions of high interest and relevance in participating countries. A 
conceptual framework for Phase II will emerge from the Phase I case studies. 

Data Collection and Analyses 

Under development as part of Phase II planning. 

Timetable 

1993 A paper outlining needs and opportunities of a civics education study was presented to 
the General Assembly (September). 

1994 An international group of experts met to discuss and formulate aims and design of the 
study. 

A proposal was drafted for approval at lEA General Assembly in August. General 
Assembly improved the study with Phase I to be identified as the lEA Civics 
Education Project. 

1995-1999 

The two-phased study will be conducted. 

1995-1996 

Phase I will be conducted - National Case Study Profiles and plan for Phase II. 
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1997-1999 

Phase II will be conducted - large-scale survey, probably of 14-year-olds, 
1998 Phase II data will be collected. 
Publications 

Civics Education Study Phase I: National case study profiles of nations and systems will be 
assembled in a publication; curriculum analysis, multi-media analysis (e.g. interactive 
interviews with students, videos) may be included. 

Civics Education Study Phase II: Large-scale survey design, methodology, sampling; 
questionnaires; and multi-media reports will be produced. 

Funding In Phase I The National Center for Education Statistics will contribute a modest amount of 
funding for the international costs; most of the funding for international costs is expected to come 
from outside the United States. 

In formation Sources 

International Association for the Evaluation of Educational Achievement QEA) Secretariat 
c/o SVO 

Sweelinclcplein 14 

2517 GK The Hague THE NETHERLANDS 
telephone: 3 1-70-346-96-79 
facsimile: 3 1-70-360-99-5 1 
e-mail: iea@svo*nl 

Judith Tomey-Purta, Ph.D., International Project Coordinator of Phase I 

and Planning Committee Chair 

Professor of Human Development 

Benjamin Building 3304, University of Maryland 

College Park, Maryland 20742 

telephone: 301/ 405-2806 

facsimile: 301/ 405-2891 

electronic mail: JT22@umaiLumd.edu 

Gordon M. Ambach, Liaison to lEA 
Council of Chief State School Officers 
One Massachusetts Ave., N.W., Suite 700 
Washington, D.C. 20001-1431 

telephone: 202/ 408-5505 

facsimile: 202/ 408-8072 

e*mail: gambach@nas.edu 

Ambach, Gordon M* 

1994 Civics Education project. Memo to William Loxley, IE A headquarters. March 21. 

International Association for the Evaluation of Educational Achievement 

1993 Activities, Institutions, and People: lea Guidebook 1993-1994. lEA, The Hague, The 
Netherlands. 
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Tomey-PurU, Judith V. 

1993 Civic Education: Need and Opportunities for a Second lEA Study. Paper prepared at the 
request of the lEA Headquarters for presentation at the lEA General Assembly. Madrid. 
September. 

1994 Needed Cross National Research on the Role of Civic Education in Democratization. Paper 
prepared for consultation session. April. American Educational Research Association. 

1994 lEA-Civics Education Study design. Presentations to Board on Intematioiud Comparative 
Studies in Education. February and May. 

Tomey, J.V., A.N. Oppenheim, and R.F. Famen 

1976 Civic Education in Ten Countries: An Empirical Study, bitemational Studies in Evaluation 
Volume 6. Stockholm: Almquist and Wiksell. New York: John Wiley & Sons. 

Tomey*>Purta» J., and J. Schwille 

1986 Civic values learned in school: Policy and practice in industrialized countries. Comparative 
Education Review, 30:30-49 



NOTE: This study summary was reviewed and edited by William Loxtey at the International 
Association for the Evaluation of Educational Achievement in The Hague on June 9, 1994, and by 
Judith Tomey-Purta and Gordon Ambach in Washington, D.C. on October 3, 1994. 
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COMPARATIVE STUDY OF TEACHER TRAINING AND PROFESSIONAL DEVELOPMENT 
PRACTICES IN APEC MEMBERS 



International Organization Asia-Pacific Economic Cooperation 
Years o f Data CoUection 1993 

Purpose The Asia-Pacific Economic Cooperation (APEC) was formed in 1989 as a new mechanism 
for multilateral cooperation among the economies of the Asia-Pacific region. Its members are 
Australia, Brunei Darussalam, Canada, Chile, the People's R^ublic of China, Hong Kong, 
Indonesia, Japan, the Rq[)ublic of Korea, Malaysia, Mexico, New Zealand, Papua New Guinea, the 
Rq)ublic of the Philippines, Singapore, Chinese Taipei, Thailand, and the United States. APEC 
aims to sustain growA, development, and improved living standards in the Asia-Pacific region and the 
world, and to promote free trade. APEC provides a forum for mmisterial discussion on a broad 
range of issues related to these goals. 

In August 1992, education ministers and senior officials from the APEC member countries met in 
Washington, D.C. to agree on activities and goals for a new APEC Education Forum. An education 
symposium was held immediately prior to the ministerial meeting to provide an opportunity for: 
exchange of ideas on designing schools for the 21st century; development of standards for curricula 
and assessment; and teacher development - standards and reform. 

The Comparative Study of Teacher Training and Professional Development Practices in APEC 
Members is one of several cooperative activities undertaken by the Education Forum. As APEC 
members develop or revise national standards in specific curriculum subjects, and standards for 
teacher education, they will also need to revise standards for teacher education. 

Organization and Management The U.S. Dq>artment of Education is coordinating this study in 
which twelve APEC countries are participating Oncluding the United States). Linda Darling- 
Hammond and Velma Cobb of the National Center for Restructuring Education at Teachers College, 
Columbia University are the principal investigators. Harold Stevenson of the University of Michigan 
has been conunissioned to write a brief literature review and is one of six members of an advisory 
panel. 

Dssign 

Participants The twelve participating APEC members are: Australia, Brunei, Canada, 
People's R^ublic of China, Hong Kong, Japan, Republic of Korea, Malaysia, New Zealand, 
Singapore, Chinese Taipei, and the United States. 

Procedures and Summary of Content The participating APEC countries have submitted ps^ers that 
describe their teacher training and professional development practices. These papers were based on a 
standard research framework. A second draft of the literature review has been completed and 
circulated to Forum members; a draft analysis overview also has been completed. 

Data Collection and Analyses The edited descriptive papers, a brief literature review, and an analytic 
overview ps^er that seeks to draw out the most important findings from a review of the member 
papers will be published in late 1994, after member approval. 
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Timtfebh 



1992 Education ministers and senior officials of APEC member countries met in 
Washington, D.C* and agreed to establish an Education Forum to engage in joint 
cooperative activities and ongoing dialogue on key education policy issues. (August) 

1993 The U.S. delegation to APEC presented a draft research framework. (May) 

1994 Twelve participating countries provided pzp&cs describing teacher traming and 
professional development practices. (January) 

A brief commissioned literature review was completed. (May) 
An analytic overview ps^er was drafted. (May) 
Members will submit connmients on the p2q)ers. (July) 
Papers, overview, and literature review will be published. (Fall) 

Piiblieations 

A volume containing papers that describe individual APEC members' teacher training and 
professional development practices, an analytic overview of the papers, and literature review 
will be published. (1994) 

Database No data base is available to the public at this time. 

Funding The U.S. Dq;)artment of Education and APEC member education ministries provide funding 
for this study. 

If^formation Sources 

Alan Ginsburg, Director 

U.S. Department of Education Planning and Evaluation Service 
400 Maryland Avenue, S.W., Room 3127 
Washington, D.C. 20202 

telephone: 202/ 401-3132 

facsimile: 202/ 401-3036 



Asia Pacific Economic Cooperation 

1992 Declaration of the APEC Education Ministerial. EFl. 

1992 Education Standards in the Asia-Pacific Region. EF2. 

1992 Proceedings of the APEC Education Ministerial. EF3. 

1993 Papers Presented at the Syn^ium on Education in the APEC Region. EF4. 
1993-4 APEC Education Forum: Summary Reports. Firstt second, thirds fourth meetings. 

1994 Curriculum Development and Achievement Standards in Mathematics Education. Final draft. 
EPS. June. 

1994 School Education Statistics in the Asia-Pacific Region. Final draft. EF6. June. 

1994 Information Technology in Education. Final draft. EF7. June. 

1994 Teacher Preparation in the Asia-Pacific Region. Publication expected. October. 

1994 Resource Allocation in Postaecondary Education. Conference proceedings. In preparation. 

1994 The APEC Education Forum. Information brochure. In preparation. 



###### 
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NOTE: This study summary was reviewed and edited by Lenore Garcia at the U.S. Dq)artment of 
Education Office of the Under Secretary, Planning and Evaluation Service in Washington, D.C. on 
July 12, 1994; database information was provided by Lenore Garcia on February 17, 1995. 
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COMPUTERS IN EDUCATION STUDY 



Iftternatioru d QreanizatUftt International Association for the Evaluation of Educational 

Achievement (lEA) 

Years o f Data Collection Stage I: 1988-1989 

Stage n: 1991-1992 

Purpose To assess to what extent, and how computers are used in education; to study the effects and 
changes that are taking place over time; and to look at how different material conditions (such 
as hardware and/or software availability) as well as immaterial conditions (such as teacher training 
facilities or support structure) affect the implementation of the use of computers by educational 
practitioners. 

In the history of education, tiie 1980's will stand out as the decade during which m&ny countries 
throughout the world introduced computers in schools on a large scale. This technological innovation 
is not only unprecedented in its scope, but also in the controversies it raised. What is the place of 
computers in the curriculum? What are the potential and actual benefits of using computers as a tool 
in the instructional process? And which strategies prove to be beneficial in implementing computers 
in educational practice? 

The Stage I survey was designed to provide answers to questions about trends in computer use, for 
example, in their distribution across subjects and d^artments. Are more scimce teachers able to 
simultaneously use more than tiie one or two computers that they typically had in 1985? Are more 
high school mathematics teachers usmg computers for classes in algebra, geometry, and trigonometry, 
instead of two-thirds of their instructional computing time going for programming and computer 
literacy classes, as in 1985? Are English classes now the major users of word processing programs, 
or is this use of computers still overwhelmingly a business education activity as it was in 1985? It 
also sought to be able to show whether there had been changes in the kinds of software and types of 
tasks students are engaged in at computers. 

The survey was intended to address questions about the effectiveness of current U.S. implementations, 
regardless of their relevance in an international context. How much staff support for schooMevel 
implementations is there and how adequately trained is it? Are schools having problems allocating 
limited computer resources among competing departments whose interest in computers has been 
piqued, and among alternative uses made possible by the burgeoning variety of software? How much 
informal staff education and training goes on among teachers within the school building, and in what 
contexts does this occur most readily? 

By the 1990s, computers have become a way of life in American schools. Ninety-nine percent of the 
elementary and secondary schools in the United States have installed computers, and 85 percent of the 
students use them during the school year. Each participating country in the Stage II Computers in 
Education Study administered an international test of practical computer knowledge, testing general 
concepts such as hardware and copy-protection, use of diskettes and software, and 2q[>plication of 
software (such as word processing, databases, and telecommunications). 

The Stage 11 survey probed questions about what students learn about computers; how students use 
computers in school; what learning about computers takes place out-of-school; sex, race, and financial 
equity; how teachers cope with computers; and computer equipment in the schools. 
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OrtaniTotion and Management 



Stage I: The international coordinator and coordinating center were located at the University of 
Twente, Enschede, The Netherlands. The U.S. national project director and coordinating center were 
located at The Johns Hopkins University, Baltimore, Maryland. Data collection was subcontracteci to 
Survey Research Associates in Baltimore, Maryland, a professional firm, selected by a competitive 
bid process. International and national steering committees provided guidance. 

Stage II: The international coordinator and coordinating center were located at the University of 
Twente, Enschede, The Netherlands. The U.S. national prcject director and coordinating center were 
located at the University of Minnesota. U.S. data collection and processing was carried out by 
national project staff at the University of Minnesota and subcontracts with Westat, Inc. International 
and national steering conmiittees provided guidance. 

Dssign 

Participants Educational systems participating in the study are as follows: 

Stage I: Austria, Belgium (Flemish), Belgium (French), Canada (British Columbia), China, 
France, Germany (Federal Republic), Greece, Hungary, India, Israel, Italy, Japan, 
Luxembourg, Netherlands, New Zealand, Poland, Portugal, Slovenia, Spain, Switzerland, 
United States. 

Stage II: Countries for which data were collected and processed in time for the lEA reports 
for Stage 11 were: (populations listed after each country) Austria (2,3), Bulgaria (2,3), 
Germany (2), Greece (3), India (3), Israel (1,3), Japan, (1,2,3), Latvia (3), Netherlands (1,2), 
Slovenia (3), Thailand (3), and the United States (1,2,3). Two countries were incomplete 
participants in that they collected some data but did not complete the study due to problems of 
schedules or resources: Hungary and Italy. 

Sample 

Stage I: Three populations of schools formed separate universes for sampling and instrument 
development: schools containing (in the U.S.) grade S, schools containing grade 8, and 
schools containing grade 12. The study data collection goal was to obtain data from 400 
schools per population. 

Stage II: Because the samples were selected using a complex probability sampling scheme, 
survey weights were needed to ensure that the sampled data could be used to make accurate 
inferences about the target populations. The U.S. samples of students and schools in Stage II 
were intended to support the following types of analysis: student*level analysis for the target 
population (what percentage of students in Sth, 8th, or 11th grade use computers extensively) 
and schooMevel analysis for each target population (what percentage of schools that offer the 
target grades use computers extensively in instruction). 

Procedures and Summary of Content The study is a two-stage survey. During the first stage (1987- 
1990)» data were collected at school and teacher level. The second stage (data collection in 1992) 
consists of a partial replication of the data collection for Stage I to be able to study changes over 
time, and the collection of data on student level for assessing students' functional knowledge and 
skills with respect to new information technologies and their attitudes towards computers. 
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The instrumentation and respondents for Stages I and II of the study consist of questionnaires for 
school principals of computer-using as well as non-using schools, technically informed persons 
(usually the computer coordinator), and teachers of computer education courses. In Stage I samples 
of computer-using and non-using teachers of mathematics, science, and mother tongue were surveyed; 
this became an international option in Stage II. 

Stage I: The United States participated along with 17 other countries and national regions in this 
study of the instructional uses of school computers. The survey provided data about how schools use 
computers to assist in traditional academic subjects as well as how they organize instruction on 
computer-specific subject-matter such as computer programming and word processing. It 
encompassed both elementary and secondary education, with a concentration on three specific 
age/grade cohorts - defined as the modal grade in each country for students aged 10 (U.S. grade 5), 
13 (grade 8), and the last year of secondary school (grade 12). 

At the teacher level, the study focused on four particular subjects: mathematics, science, English 
(-mother tongue"), and computer education. Although itself only a survey of school and teacher 
practices, the survey was also preliminary to the effort planned for 1992 that would measure the 
effect of computer-based educational experiences on student achievements in computer-specific 
subject-matter ^programming and computer literacy) and in traditional subjects (such as mathematics 
and science) where computers are used. 

Besides its comparative fiinction, the survey also served an important domestic data collection 
function for the United States, extending time-series data previously collected in 1983 and 1985 from 
national samples of schools and teachers in this countty. By 1989 most U.S. schools had enough 
computers in single locations that entire classrooms of students could be served at one time - a 
situation that did not prevail during previous school surveying. 

Stage II: The international survey tested 69,000 suidents in grades 5, 8, and 11 in 2,500 schools on 
practical computer knowledge of general concepts such as the use of diskettes and software and the 
general features of common application software - word processing, databases, and 
telecommunication. Computer programming was not tested. The U.S. portion of the study included 
11,284 students in 573 schools. 

Data Collection and Analyses 

Stage II: Because funding was not provided until Sq)tember 1992, the United States was unable to 
conduct a full scale pilot testing within the time schedule of the international coordinating center. 
Nevertheless, just before the national project coordinators meeting in September over 150 students in 
tiiree different population groups were pilot tested, and that dato was taken to the Sq)tember NPC 
meeting. Thus, the U.S. data were combined quantitatively and qualitatively with the data from other 
countries at the meeting and in subsequent analyses. The U.S. pilot daU was used extensively in the 
instrument development at the September NPC and international steering committee meetings. 

The United States administered optional performance tests in grades 8 and 11. About 1 10 schools 
chose to participate and over 1,500 students completed the word processing performance test. Data 
from the follow-up surveys of students* test administrators were coded and assembled in data files for 
analysis. 

At the April 1994 meetings of the American Educational Research Association Wayne Welch 
presented a paper on a longitudinal analysis of daU using organizational factors to predict growth in 
the quantity and quality of educational computer utilization. 
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The national project team is prq>aring public data files and extensive associated documentation: 
coding, analysis, and r^rting of performance assessment data; coordination with the lEA chair in 
preparation of the collection of national context articles describing the features of the educational 
system in each country; more detailed analyses of the data for academic journals; and preparation of a 
technical memo on equity issues arising from the national report. 

An assessment of progress toward computer-related standards in mathematics and science education 
has been proposed. The data from the lEA study would be used to determine the extent to which 
U.S. students and schools are meeting the computer-related standards specified by the National 
Council for Teachers of Mathematics standards and the American Association for the Advancement of 
Science benchmarks for science literacy. 

Production of a 60-minute video program has been proposed, using results from the international 
study. The video would be broadcast on national television in the United States and other countries. 
English and German language versions would be produced. 

Timetable 

Stage I: 

1988- 

1989 Data was collected. 

Stage II: 

1991- 

1992 Data was collected. 

National Project Coordinators met in Tokyo. (September) 
National Steering Conmiiitee met. (November) 

1992- 

1993 Data was processed. 
1993- 

1994 Reports were prepared. 

1993 Presentations were made on national and international lEA Computers in Education 
Study activities at the American Educational Research Association meetings. 

National Research Coordinators and Internationai Steering Conmiittee met in 
Washington, D.C. (September) 

Study results were released simultaneously at lEA Headquarters in The Hague with 
the report Schools, Teachers, Students and Computers: A Cross-National Perspective, 
and in Washington, D.C., with the U.S. report. Computers in American Schools, 
1992: An Overview. A press release was issued in Washington, D.C. (Dec ^nber) 

1994 A presentation was made at the American Educational Research Association meeting. 
(April) 

Public data files for the United Sutes with codebooks were completed. (July) 
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U*S. project staff are cooperating with the international coordinating center to analyze 
data and write materials for several international reports: 

o the performance assessment rq[>ort 

o an ^ited collection of psqpers on national context of computer education 

(Cross National Policies on Computers in Education) 
o a short report on the student assessment (Functional Computer Literacy: The 

First International Assessment) 
o the final collection of technical articles called the volume." 

A video program using results from the international study, to be broadcast in the 
United States and other countries, in English and German language versions is 
proposed* 

An assessment of progress toward computer-related standards in mathematics and 
science education is proposed. (United States) 

PutfUcatiQns 

International: 

Schools, Teachers, Students, and Computers: A Cross-National Perspective. International 
Association for the Evaluation of Educational Achievement, University of Twenty, Enschede, 
The Netherlands. (1993) 

The Use of Computers in Education Worldwide: Results from the lEA Computers in 
Education Survey in 19 Education Systems. Oxford: Pergamon Press. (1991) 

Additional forthcoming international publications: the performance assessment report, an 
edited collection of papers on national context of computer education (Cross National Policies 
on Computers in Education), a short report on the student assessment (Functional Computer 
Literacy: The First International Assessment), and the final collection of technical articles. 
(1994) 

National: 

Computers in American Schools, 1992: An Overview. lEA Computers in Education Study, 
University of Minnesota, Minneapolis, Minnesota. (1993) 

Stage I: Funding for U.S. participation in Stage I was provided by the National Science 
Foundation. 

Stage n: Funding for U*S* participation in Stage II was provided by the National Science 
Foundation after September 1992* (Prior to that time the work on Stage 11 was 
provided on a volunteer basis and supported with a small grant from NSF for a 
meeting to review the potential and problems of performance testing*) A request has 
been made (1994) to the National Science Foundation for a small funding supplement 
for purposes of assessing progress toward computer-related standards in mathematics 
and science education* 
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Information Soun^es 



Dr. Hans Pelgrum, International Coordinator 
University of Twente Department of Education 
Centre for Applied Educational Research 
P.O. Box 217 

7500 AE Enschede THE NETHERLANDS 
telephone: 31-53-893-593 
facsimile: 31-53-315-099 
e-mail: topelgrm®utwente.nl 

International Association for the Evaluation 
of Educational Achievement (lEA) Secretariat 
do SVO 

Sweelinclcplein 14 

2517 GK The Hague THE NETHERLANDS 
telephone: 3 1-70-346-96-79 
facsimile: 31-70-360-99-51 
e-mail: iea@svo.nI 

Stage I: 

Dr. Henry Jay Becker, National Project Director 
^ Johns Hopkins University 

Center for Social Organization of Schools 
3505 North Charles Street 
Baltimore, Maryland 21218 
telephone: 301/ 338-7570 

Stage II: 

Dr. Ronald E. Anderson, National Project Director 
Department of Sociology 

909 Social Sciences Bldg., University of Minnesota 
267 - 19th Ave., South 
Minneapolis, Minnesota 55455-0412 

telephone: 612/ 624-9333 

facsimile: 612/ 624-4586 

e-mail: rea@vx.cis.unm.edu 

Gordon M. Ambach, Liaison to lEA 
Council of Chief State School Officers 
One Massachusetts Ave., N.W., Suite 700 
Washington, D.C. 20001-1431 

telephone: 202/ 408-5505 

facsimile: 202/ 408-8076 

e-mail: gambach@nas.edu 



Ambach» Gordon M. 

1990 Memonuiduin to selected adviflon to the U.S. delegate to the lEA General Assembly re Phase 
II of the Computers in Education Study of lEA. July 5« 
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Anderson, Ronald E. 

1990 lEA Conq>uters in Education Study plans for Phase 2. Presentation to the Board on 
International Comparative Studies in Education. September. 

1991 lEA Con^uters in Education Study. Status report prepared for the Board on International 
Comparative Studies in Education. January. 

1992 lEA Con4>uters in Education Study. Status report prepared for the Board on International 
Comparative Studies in Education. January. 

1993 lEA Computers in Education Study. Status reports prepared for the Board on International 
Con^Mirative Studies in Education. January and June. 

1993 lEA Con4>uters in Education Study dissemination activities. Presentation to the Board on 
International Conq)arative Studies in Education. June. 

1993 International Survey on Computers in Education Reveals U.S. Strengths and Potential. Press 
release. December 14. lEA Computers in Education Study, University of Minnesota, 
Minneapolis, Minnesota. 

1994 lEA Computers in Education Study. Status reports prepared for the Board on International 
Con^Murative Studies in Education. January and May. 

Anderson, Ronald E., Editor 

1993 Computers in American Schools, 1992: An Overview. A national report from the 

International lEA Computers in Education study. lEA Computers in Education Study, 
University of Minnesota, Minneapolis, Minnesota. 

Anderson, R.E., and B. Collis 

1993 International assessment of functional computer abilities. Studies in Educational Evaluation. 
Special issue 19(2). Oxford: Pergamon Press 

Anderson, Ronald E., and Larry Suter 

1993 lEA Computers in Education Study plans for releasing data. Presentation to the Board on 
International Comparative Studies in Education. November. 

Becker, H.J. 

1993 Computer experience, patterns of computer use, and effectiveness. An inevitable sequence or 
divergent national cultures? Studies in Educational Evaluation. Special issue 19(2). Oxford: 
Pergamon Press. 

Becker, Henry Jay 

1988 lEA Computers in Education Study (1989): Summary of United States Designs and Plans. 
Objectives, san4>ling plan, and questionnaire. October. Presentation to the Board on 
International Comparative Studies in Education, November. 

1989 lEA Computers in Education Study. Status report prepared for the Board on International 
Con4)arative Studies in Education. July. 

1989 lEA Computers in Education Study. Status report presented to the Board on International 
Comparative Studies in Education. September. 

1990 Computer Use in United States Schools: 1989. An Initial Report of U.S. Participation in the 
lEA Computers in Education Survey. Paper presented at the 1990 meetings of the American 
Educational Research Association. April. Center for Social Organization of Schools, The 
Johns Hopkins University, Baltimore, Maryland. 

Brummelhuis ten, A.C.A., and Tj. Plomp 

1993 The relation between problem areas and stages of computer implementation. Studies in 
Educational Evaluation. Special issue. 19(2). Oxford: Pergamon Press. 

International Association for the Evaluation of Educational Achievement 

1993 Activities, Institutions, and People: lEA Guidebook 1993-1994. lEA, The Hague, The 
Netherlands. 
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Janssen Reineo, I.A.M*, and Tj. Plomp 

1993 Staff development as a condition for computer integration* Studies in Educational Evaluation. 

Special issue* 19(2). Oxford: Pergamon Press. 
1993 Gender differences in computer use with emphasis on mathematics education. International 

Journal of Educational Research. Accepted for publication. Oxford: Pergamon Press. 
1993 Some gender issues in educational computer use: Results of an international conq)arative 

survey. IntemationalJoumal of Educational Research. Acc^ted for publication. Oxford: 

Pergamon Press. 



Attitudes of school principals and teachers towards computers: Does it matter what they 
think? Studies in Educational Evaluation. Special issue. 19(2). Oxford: Pergamon Press. 
International research on computers in education* PROSPECTS. 18(5). UNESCO, Paris. 

Pelgrum, W.J., I.A.M* Janssen Reinen, and Tj. Plonq>, Editors 

1993 Schools, Teachers, Students and Computers: a Cross-National Perspective. lEA-Comped 
Study Stage 2. International Association for the Evaluation of Educational Achievement, 
University of Twente, Center for Applied Educational Research, Enschede, The Netherlands. 

Pelgrum, W.J., and Tj. Plomp 

1991 The Use of Computers in Education Worldwide: Results from the lEA Computers in Education 

Survey in 19 Education Systems. Oxford: Pergamon Press. 
1993 The use of computers in 18 countries. Studies in Educational Evaluation. Special issue. 
19(2). Oxford: Pergamon Press. 

Pelgrum, W.J., and Tj. Plomp, Editors 

1993 Computers in Education I: Implementation of an Innovation in TWenty Countries. Oxford: 
Pergamon Press. 

1993 Schools, Teachers and Computers: a Cross-National Perspective. Preliminary report, lEA- 
Comped Study Stage 2. International Association for the Evaluation of Educational 
Achievement, University of Twente, Center for Applied Educational Research, Enschede, The 
Netherlands. 



Pelgrum, W.J* 
1993 

1993 



Pelgrum, Willem J., and Tjeerd Plomp 

1991 The Use of Computers in Education Worldwide: Results from the lEA Computers in Education 
Survey in J9 Educational Systems. 



Wolf, R.M. 

1993 The role of the school principal in computer education. Studies in Educational Evaluation. 
Speci li ^e. 19(2). Oxford: Pergamon Press. 



NOTE: This study summary was reviewed and edited by Ronald Anderson at the University of 
Minnesota on July 8, 1994. 
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CROSS-NATIONAL STUDIES OF ADULT UNDERSTANDING OF SCIENCE 



International Organization Chicago Academy of Sciences 

Years o f Data CoUeetion 

NSF-Nm national survey of public understanding of science: 1992 and 1994 
Co-ordinated cross-national data sets at International Center: ongoing 
Data set archives cooperative arrangements: ongoing 

Purpose The Chicago Academy of Sciences International Center for the Advancement of Scientific 
Literacy has been selected by the National Science Foundation and the National Institutes of Health to 
design and conduct national surveys of the United States in 1992 and ISKKI concerning the public 
understanding of science. The International Center developed one questionnaire to continue trend 
measures and a second questionnaire to measure the public understanding of biomedical and 
behavioral science concepts. In both of these studies, attention is being focused on both the 
preservation of trend measures within the United States and the establishment and continuation of 
comparative international indicators of the public understanding of science. 

Ihe International Center provides a point of coordination for and dissemination of cross-national 
study data, as well as an active research program and a comprehensive archive of data sets relevant to 
student and adult learning about science, mathematics, and technology. Center director Jon Miller 
believes that the interface between the two population segment data sets is extremely important, that 
the education of young people begins to play out in adult behaviors and adult choices. 

Over the last several decades, the influence of science and technology on the lives of citizens and 
consumers has become apparent. The number of public policy issues involving some aspect of 
science or technology has been increasing sharply over recent decades, and there is every reason to 
believe that it will continue to increase at an accelerating rate in the 21st century. In the next century 
citizens will need to be significantly better informed about scientific and technological concepts in 
order to fulfill roles in government, the marketplace, conununication, and transportation. 
Governments in most industrialized nations are making concerted efforts to address the issue of 
pervasive scientific illiteracy . 

One of the difficulties that all of the governments and organizations who are working in this field face 
is the absence of a point of focus and coordination for these efforts, especially a point of coordination 
that brings together the ideas and experiences of researchers and educators from countries around the 
world. While there are important differences in the educational systems of various countries, there 
are also commonalities. There are more areas in which researchers and educators can cooperate and 
learn from each other than there are areas of unique differences that would make cross-national 
comparisons inapplicable. 

In response to a consensus among scholars and educators who have been working in this field, the 
International Center for the Advancement of Scientific Literacy was established at the Chicago 
Academy of Sciences in 1991. To provide a continuing linkage with working scholars and policy 
makers who utilize studies of scientific literacy , the International Center has formed an International 
Program Council* Through an annual meeting of the International Program Council and periodic 
consultations throughout the year, the Council provides continuing program guidance and assures both 
the international character of the programs of the Center and the utility and quality of its programs 
and products. 
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The primary program focus of the International Center is the conduct and coordination of cross- 
national research regarding the development and maintenance of scientific literacy. To date, the 
most extensive work in this area has been done in the United States. There is emerging a strong 
interest on this topic in Europe, Japan, and other countries. 

In regard to school-aged populations, the International Center will serve as the design and analysis 
center for the Longitudinal Study of American Youth. This national study of the development of 
attitudes toward and competence in science and mathematics among middle school and high school 
student populations is being conducted primarily in the International Center to utilize more fully 
comparable national and international data s^ and staff familiar with other educational systems. 

Parallel to the conduct of new research, a second major research focus of the International Center is 
the development of a research archive of data being collected about the public understanding of 
and attitudes toward science and techiiOlogy in countries throughout the world. About 60 percent 
of the known national studies of scientific literacy among adults, but significantly fewer of the studies 
concerning the development of an interest in and knowledge of science and mathematics during the 
pre-adult years have been collected. The International Center has begun to make these materials 
available for use by interested scholars and analysts. 

In addition to providing data access, the International Center seeks to become a place that scholars 
and students can visit in order to learn more about these materials and to conduct research on them. 
To provide a forum for the exchange of research and the sharing of ideas, the International Center 
sponsors an annual research conference. The first meeting was held in 1992 in Tokyo and was jointly 
sponsored by the Japan Science Foundation, the National Institute of Science and Technology Policy 
(Japan), and the National Institute of Educational Research (Japan). 

In 1993, the International Center hosted an International Conference on the Public Understanding of 
Science and Technology. The conference attracted more than 155 participants from 22 countries. 
Prior to the 1993 International Conference, the International Center conducted a two*day workshop on 
structural equation modeling. 

A second major program focus of the International Center is the development of scientific literacy 
during the common school years. Whilie there are several other centers that focus on important 
parts of science and mathematics education, most of the existing centers are more interest^ in the 
education of future scientists, mathematicians, and engineers than they are in the development of 
scientific literacy among citizens who will not work professionally in science or technology. The 
International Center focuses on the design and implementation of curricula and programs to serve the 
needs of young people who will be citizens of an increasingly scientific and technological world but 
who will not be professional scientists. 

A third major program focus of the International Center is the communication and dissemination of 
scientific information to broader adult audiences, often referred to as information science 
education. Over the course of the last two decades, there has been a substantial growth in the 
conununication of scientific information to adult populations in the United States and other industrial 
countries through the expansion of science and technology museum programs, natural history museum 
programs, zoos and aquaria, and through the growth of science television and science magazines. 

There is a good deal to be learned from the experiences of each nation in this regard, and many 
science communicators are particularly eager to share their own experiences and learn about the work 
that is occurring in other nations. While each informal education channel has its own professional 
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organizations, many scholars in the field and practicing professionals have indicated that it would be 
usefiil to have a periodic international meeting to bring together the different parts of the informal 
science education chain, thus achieving greater cooperation and greater impact. The International 
Center will establish an exploratory planning conunittee in this area and examine the needs and 
potential benefits of an international meeting in this area. The International Center will continue to 
collect relevant data for its archive and relevant publications for its library. 

Oivanization and Manafemeni 

Co-ordinated cross-national data sets; To coordinate the design of questionnaires and the 
definition of terms and constructs in the emerging set of cross-national studies and to foster an 
annual discussion among the principal data collectors, the International Council for the 
Comparative Study of the Public Understanding of Science and Technology was formed and 
has held its meetings in London (1990), Tokyo (1992), and Chicago (1993). The Council is 
comprised of two representatives from each active data collecting country. The Council has 
been a useful forum for discussing problems or getting to know the other people active in this 
field. The 1993 meeting in Chicago focused on two substantive problems. The Chicago 
Academy of Sciences director, Jon Miller, was active in forming the Council and continues to 
participate in it. 

Recognising the inherent limitations of the Council structure, an International Center for the 
Advancement of Scientific Literacy was created at the Chicago Academy of Sciences. The 
International Center provides a point of coordination and dissemination that has a central staff, 
an active research program, and a comprehensive archive of data sets relevant to student and 
adult learning about science, mathematics, and technology. 

To create an environment for a more substantive discussion of the measurement of the public 
understanding of science and technology, the Chicago Academy of Sciences and the London 
Science Museum have agreed to sponsor an annual international research conference. The 
first conference was held in Tokyo (1992); the second in Chicago (1993). The 1994 
conference will be held in London, and the 1995 conference in Beijing (in cooperation with 
the China Association for Science and Technology). Miller believes that a conference seuing 
with formal papers and opportunities for dialogue will improve the quality of coordination. 
The meetings of the International Council have now been linked to diese research conferences. 

Data set archives cooperative arrangements: To provide for improved data sharing, the 
International Center has created an archive of data sets. The archive is available to interested 
scholars for short-term visits, longer-term visiting scholar sq)pointments, and for the 
distribution of data sets by tsqpe or wire. The archive presently holds approximately ISC data 
sets and has identified an additional 100 daU sets for acquisition over the next two years. 
The International Center is a member of the ICPSR and has established cooperative 
arrangements with the ESRC Archive (U.K.), the Steinmetz Archive (Netherlands), the 
Central German Archive (Koln), the Danish Archive (Odense), and the Swedish Archive 
(Gothenberg). The International Center is presently seeking corporate and government 
support for the expansion of the International Center archive and the support of visiting 
scholars. 
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Design 



Participants 

NSF-NIH national survey of VS. public understanding of science: 1992 and 1994: 
International Center conducted the study with the U.S. public. 

Co*K>rdinated cross<*natlonal study data sets in International Center: 

Britain, Bulgaria, Canada, China, European Community, Japan, Spain, New Zealand, 
United States 

Data set archives cooperative arrangements: 

ESRC (United Kingdom), Steinmetz (Netherlands), Central German Archive, Danish, 
Swedish 

Procedures and Summary of Content 

Co-ordinated cross*national study data sets In International Center: The coordinated 
cross-national study of the public understanding of science and technology at the national level 
is a relatively recent phenomenon, beginning with a joint study between die United States and 
Britain in 1988. The origin of this work comes from the National Science Board. Beginning 
in 1972 and continuing biennially the NSB has published Science and Engineering Indicators 
that includes a chapter on the public understanding of science and technology. The data for 
the 1972, 1974, and 1976 reports were collected by adding 20 questions to an omnibus survey 
conducted by the Opinion Research Corporation; since 1979 the National Science Foundation 
has sponsored a national probability sample study devoted to measuring the public 
understanding of science and technology. For the 1979 study in-person interviews were 
conducted by the Institute for Survey Research at Temple University. 

Within the past several years the United Kingdom, the European Community, Canada, Japan, 
New Zealand, China, Spain, and Bulgaria have conducted similar surveys; Korea, Bdgium, 
and Germany are also planning surveys. While the United States has a 20-year time series, 
only the European Community has successfully completed a second cycle of data collection to 
date. There are two continuing efforts to coordinate the design and collection of these cross- 
national studies and to disseminate the resulting data sets. 

Data Collection and Analyses 

Co-ordinated cross-national study data sets in International Center: The International 
Center has a small but growing set of studies and data sets, and an expanding network of data 
collectors and analysts in an increasing number of countries. It has tsdcen some initial steps to 
improve coc lination and dissemination. The current inventory of cross-national studies and 
data sets is as follows: 
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National Studies of the Public Understanding of Science and Technology 



Year 


Country 


Principal Investigator 


N 


RA 


DA 


1957 


US 


Davis, Robert 


1,999 


Y 


Y 


1972 


US 


NSB 


1,500 


Y 


Y 


1974 


US 


NSB 


1,500 


Y 


Y 


1976 


US 


NSB 


1,500 


Y 


Y 


1979 


US 


Piewitt, K. and Miller, J. 


1,500 


Y 


Y 


1981 


US 


Miller, J. 


3,200 


Y 


Y 


1983 


US 


Miller, J. 


2,000 


Y 


Y 


198S 


US 


Miller, J. 


2,000 


Y 


Y 


1088 


US 


Miller, J. 


2,000 


Y 


Y 


1988 


Britain 


Thomas, G. and Durant, J. 


2,000 


N 


Y 


1989 


EC 


Gabolde, J. 


12,000 


N 


Y 


1989 




Einsiedel 


2,000 


Y 


Y 


1990 


us 


Miller, J. 


2,000 


Y 


Y 


1990 


New Zealand 


Billingham, G. 


2,000 


Y 


N 


1991 


Japan 


Nagahama, H. 


1,500 


Y 


Y 


1992 


China 


Liu, W. 


5,000 


Y 


* 


1992 


EC 


Gabolde, J. 


13,000 


N 


* 


1992 


Spain 


Pardo, S. 


7 


N 


N 


1992 


Bulgaria 


Unknown 




N 


N 


1992 


us 


Miller, J. 


2,000 


I 


I 


1992 


us 


Miller, J. 


3,200 


I 


I 



Number of cases 
Report available 
Data available 

In process 

Expect to receive summer 1993 
NSB = National Science Board 



Legend: N 
RA 
DA 
I 
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Timetable 



1994: NSF-Nm national survey of U.S. public understanding of science: 
Data was colleted. 

1994: The International Center for the Advancement of Scientific Literacy-London Science 
Museum convened an international research conference in London. 
The International Center for the Advancement of Scientific Literacy-International 
Council for the Comparative Study of the Public Understanding of Science and 
Technology met. 

199S: The International Center for the Advancement of Scientific Literacy-London Science 
Museum convened an international research conference in Beijing. 

Funding 

National survey of U.S. public understanding of science: The National Science Foundation 
and National Institutes of Health provide funding. 

Information Sources 

Jon D. Miller, Vice President 
The Chicago Academy of Sciences 
2001 North Clark Street 
Chicago, Illinois 60614 

telephone: 312/549-0606 

facsimile: 312/549-5199 



The Chicago Academy of Sciences 

1993 The International Center for the Advancement of Scientific Literacy: A Statement of Purpose 
and Program. June. The Chicago Academy of Sciences^ Chicago^ Illinois. 

Miller, Jon D. 

1992 The Public Understanding of Science & Technology in the United States. 1990. Report to the 
National Science Foundation. Analytic report accompanying the 1990 survey of the public 
understanding of science and technology, sponsored by die Science and Engineering Indicators 
Program of the Division of Science Resources Studies, National Science Foundation. Hard 
copy tabulations of data from the entire series (1979 through 1990) may be ordered for a 
nominal charge from the International Center for the Advancement of Scientific Literacy. 
Data disks are also available from the International Center. Washington, D.C.: Division of 
Science Resources Studies, National Science Foundation. 

1993 Cross-national Studies of Adult Understanding of Science. Memorandum and presentation to 
the Board on International Comparative Studies in Education. June 18. 

1993 Science and technology: Public attitudes and public understanding. Science and Engineering 
Indicators . National Science Foundation, Wa^iington, D.C. 



NOTE: This study sunmnary was reviewed and edited by Veronica Muiloz for Jon Miller at the 
Chicago Academy of Sciences on July 8^ 1994. 
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EDUCATION INDICATORS PROJECT (INES) 



Intenuaional Organhation Organisation for Economic Co*operation and Development 

Years o f Data CoUection Phase 1: 1988*1989 

Phase 2: 1990-1991 
Phase 3: 1992-1996 

Purpose Increased need for information on education led the members of the Organisation for 
Economic Co-operation and Development to initiate the International Education Indicators Project in 
an effort to create a set of comparative international education indicators that r^resent the key 
features of education systems* The project was launched via two conferences, hosted by the 
government of the United States (1987) and the government of France (1988)* The project went from 
the planning phase to the production phase following a conference on the project hosted by the 
government of Switzerland in 199 L 

The International Educational Indicators Project is an effort of the Organisation for Economic 
Co-operation and Development-Centre for Educational Research and Innovation to develop a system 
of indicators for cross-national comparisons in education for the use of policy makers, consumers, and 
''third parties** like private industry* To achieve this purpose, INES is: 

o developing, collecting, analyzing, and offering a preliminary interpretation of a set of 
key indicators for international comparisons 

o providing a forum for international cooperation and the exchange of information about 
methods and practices of devdoping and using educational indicators for national 
policymaking and managing education systems 

o contributing to evaluation methodology and practice to develop more valid, reliable, 
and comprehensive indicators, and to gain a better understanding of their use in 
policymaking* 

The task of INES Project Network A - Educational Outcomes is to: 

o Develop indicators of student performance outcomes* 

o Develop a comprehensive analytic frame for educational achievement outcomes* 

o Develop specific criteria and standards for constructing and evaluating indicators of 

educationsd achievement* 
o Apply the frame, criteria, and standards to existing international surveys* 
o Suggest how these principles can be used by OECD to obtain achievement data on a 

regular basis* 

Or^anizaiion and Management The INES Project is managed by the Centre for Educational 
Research and Innovation within the Organisation for Economic Co-operation and Development* The 
work of the project is being carried out by a consortia of countries comprising a Technical Group and 
four Networks: 

o Network A: Educational Outcomes 
(chaired by the United Sutes) 
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Network B: Education and Labor Market Destinations 

(chaired by Sweden) 

Network C: Features of Schools 

(chaired by the Netherlands) 

Network D: Attitudes and Expectations 

(chaired by the United Kingdom) 



Using a network structure as a basic organizing framework, the first Phase of the INES Project 
explored the feasibility of developing and rq[)orting comparable indicators concerning the education 
systems of participating countries. The network structure had five domains: student flows, student 
outcomes, fimctioning of schools, costs and resources, and attitudes and expectations. Each network 
had a lead country that coordinated the contributions of members, supported the theoretical and 
technical work required, and produced the network report. 

At the end of the first phase in 1989, Network 2, Student Outcomes, was split into two groups: 
Network A, Student Achievement Outcomes and Network B, Education and Labor Market 
Destinations. The United States was the lead country for Network 2, and continues in that role for 
Network A. Network A (now called Educational Outcomes) is responsible for the development and 
preparation of educational outcome indicators. 

Phase 3 project organization continues along the same lines as the previous two phases, with the 
exception that the two former technical groups have been combined, and the leadership of the network 
on School Processes has been passed from France to the Netherlands. Scotland has assumed 
responsibility for the network investigating indicators of attitudes and expectations. Network A 
continues to be chaired by Gary Phillips of the National Center for Education Statistics in the United 
States. 

Design Following an exchange of preliminary information on definition of fields, concepts, methods 
of work, and data availability, networks started working in January 1987 to establish a working plan 
for the definition of a limited set of indicators. These plans were then further refined and endorsed 
by a scientific advisory group, which in Phase 2 was reconfigured into a consultative group. In Phase 
3 the consultative group was replaced by a policy review advisory group, chaired by the chairman of 
the CERI Governing Board. In Phase 2, networks and technical groups conducted further work with 
the aim of selecting some trial indicators, establishing data sources, and undertaking some 
experimental indicator calculations. The results, analyses, and requirements for further work are 
published in Education at a Glance. 

Participants 

Eighteen countries participated in Phase 1 of the INES Project Network A - Educational 
Outcomes: Australia, Belgium, Canada, Denmark, France, Germany, Italy, Japan, 
Luxembourg, The Netherlands, New Zealand, Norway, Portugal, Spain, Sweden, 
Switzerland, the United Kingdom, and the United States. 

Seventeen countries participated in Phase 2 of the INES Project Network A - Educational 
Outcomes: Belgium, Canada, Denmark, Finland, France, Germany, Italy, Japan, 
Luxembourg, The Netherlands, New Zealand, Norway, Spain, Sweden, Switzerland, the 
United Kingdom, and the United States. 
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Countries registered for Phase 3 (1992-96) are: 

Network A Network B Network C Network D Technical 

Group 



Australia 


? 


? 


? 


? 


? 


Auitria 


X 


X 


X 


— 


X 


Belgium 


X 


X 


X 


X 


X 


Canada 


X 


X 




X 


— 


Denmark 


X 


X 


X 


X 


X 


Finland 


X 


X 


X 


X 


X 


France 


X 


X 


X 


X 


X 


Germany 


X 


X 


— 


— 


X 


Greece 


_ 


— 




— 


— 


Iceland 


— 





— 


— 


— 


Ireland 


X 


X 


X 


— 


X 


Italy 


X 


X 


X 


X 


X 


Japan 




— 


— 


• — 


X 


Luxembourg 


? 


? 


? 


? 


? 


Netherlands 


X 


X 


X 


X 


X 


New 


X 


X 






A 


Zealand 












Norway 


X 




X 




X 


Portugal 


X 


X 


X 


X 


X 


Spain 


X 


X 


X 


. X 


X 


Sweden 


X 


X 


X 


X 


X 


Switzerland 


X 


X 


X 


X 


X 


Turkey 


X 


X 


X 


X 




United 


X 


X 


X 


X 


X 


Kingdom 












United 


X 


X 


X 


X 


X 


Sutet 












ToUl 


19 


18 


16 


13 


19 



Procedures and Summary of Content In Phase 1 the INES Project Network A developed a 
conceptual framework in which education outcomes were seen as a function of resources, environment 
(including demographic, economic, and social aspects), and structural processes* Work was 
organized around three areas: completion of secondary education, cognitive achievement and other 
assessment outcomes, and activity following completion of secondary education* A list of indicators 
in each area was generated, discussed, and refined* 

Phase 2 INES Project Network A aims were to develop a comprehensive analytical frame for 
educational achievement outcomes; develop specific criteria and standards for constructing and 
evaluating indicators of educational achievement within the general template established for the INES 
Project; and apply the template, criteria, and standards to existing international surveys to test the 
extent to which existing data and procedures meet these criteria and standards* 

The INES Project entered into its third and final phase at the beginning of 1993* This phase, 
scheduled to continue for five years, aims at establishing an organizational framework allowing for 
the regular production of a set of international education indicators* It includes continuation of 
ongoing conceptual work and regular publication of the calculated indicator set as Education at a 
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Glance. At the end of this phase it is anticipated that the indicators function will become part of the 
regular responsibilities of the Education Division of OECD* 

During Phase 3 the second edition of Education at a Glance has been published (1993), and a 
completely revised edition will be published in 1994. A state-of-the-art report on the results achieved 
in implementing an effective international information system on education statistics will be an 
additional product. By the tune the INES Project is completed in 1996, the Organisation for 
Economic CoK>peration and Development and its Member countries should have succeeded in setting 
up an international database containing up-to-date and relevant education statistics and indicators; a 
fUUy computerized on-line network for collecting and disseminating the data; and cost-effective 
procedures for processing the data. 

Data Collection and Analyses In Phase 2 INES Project Network A surveyed member countries in an 
attempt to identify existmg data from national assessments or examinations that might serve as 
outcome indicators. The survey produced much valuable information about the variety of assessment 
structures and practices, and allowed some conclusions to be drawn about commonalities. It also 
demonstrated that it would be extremely difficult, if not impossible, to d^end on data produced from 
these examinations or assessments to produce comparable indicators, as they vary widely in purpose, 
ages tested, subjects tested, and forms of testing. 

Network A proposed a set of indicators (multiple comparison of mean achievement scores, 
international comparative distribution of achievement scores, learning/teaching ratio, and between 
school and between classroom variation in achievement scores) that can provide information about the 
productivity of the system relative to other countries, the effectiveness with which the curriculum has 
been learned, and how equitably achievement is distributed within a country. These indicators were 
calculated on a trial basis using data firom the lEA Second International Mathematics Study and the 
mathematics portion of the Second International Assessment of Educational Progress. The results 
were published in the first edition of Education at a Glance. 

In Phase 3 each of the INES Project networks and the technical group are involved in data assembly 
and preparation of indicators for the third edition of Education at a Glance. The technical group is 
piloting the use of a new data collection form for finance data. This form is also being used as the 
basis for a revised OECD/UNESCO/EUROSTAT pilot of the joint survey. This form is an 
outgrowth of the Expenditure Comparability Study conducted for the National Center for Education 
Statistics. 

Current long-term activities of the INES Project include writing and editing chapters for several 
technical volumes that the project will publish in 1995, piloting data collections to measure cross- 
curricula competencies, and examining the use of national assessments for international comparisons. 
Each network is pr^aring books on their work to date and its implications for future international 
comparison work. These books are intended to compile and make more publicly available the 
methodological contributions achieved in Phase 3 of the INES Project. 

The second edition of Education at a Glance was published in December 1993. It contains 38 
indicators organized by 1) cost, resources, and school processes; 2) contexts of education; and 3) 
results of education. National Center for Education Statistics representatives to the project's networks 
and technical group were responsible for providing information about the United States and pr^aring 
the student outcome indicators. NCES is involved in developing ideas for indicators and data 
collection strategies for the indicators to be published in the third edition of Education at a Glance. 
These ideas include a survey on attitudes toward education, a description of the incorporation of 
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national goals into educational systems, and the measurement of the educational attainment of workers 
in different occupations and industries. 

The INES Project attitudes survey is a public opinion survey that includes questions on subjects that 
young people study in secondary school, qualities that young people may have developed by the end 
of their compulsory schooling, the importance of areas for schools to emphasize in order to achieve 
their goals (e.g., career advice), how much responsibility the school should have (compared with the 
home) for the personal and social development of young people, the importance of certain decisions 
being made at the school leveU and the respect for and earnings of secondary school teachers. About 
ten OECD countries will be participating in the survey* 

For the second edition of Education at a Glance, Network A prepared indicators for reading literacy, 
mathematics, and science for 13/14*year-olds. Three types of indicators were pr^ared: comparison 
of means (multiple^omparison chart); distribution of scores; and between and within school variance. 
In addition, Network A prq^ared a brief cross-subject-based introduction to its section. As with 
indicators presented in the first edition of Education at a Glance, all indicators used the standards that 
Network A has adopted for its work. 

Network A remains committed to calculating the indicators already developed, and the second edition 
of Education at a Glance presented an expanded set using science data from the International 
Assessment of Educational Progress study and indicators of reading literacy from the International 
Association of Educational Achievement Reading Literacy Study at two different age populations. 
Network A has also explored some new indicators of cognitive achievement, including attenq)ts to 
measure the '*value added"* to student learning by schooling, and a measure of change over time. 

At a very early stage within Network A it was quite clear that the Network needed a set of technical 
standards by which to judge the adequacy of education outcome indicators. It was felt that such a set 
of standards were needed because most outcomes data available to the Network were based on 
surveys that have estimable sampling error and outcome data usually involve the administration of 
tests, which also involve estimable measurement of error. 

Because of these two sources of error it was felt that users of the international outcomes indicators 
should have some sense of the technical quality of the indicators. Network A has drafted standards 
for international indicator data, for the purpose of directing a reader to the fact that a standard is not 
being met and therefore to results that ought to be interpreted with caution. Network A recommends 
that future international assessment studies be conducted with these standards in mind, and that other 
indicator projects consider technical standards as part of their reporting practices. 

In addition. Network A has significantly expanded its scope to explore ways in which indicators of 
non cuniculum bound outcomes might be developed. In the first instance, this developmental work 
concentrated on identifying suitable instrumentation and data sources for measuring socio-cultural 
knowledge and skills, such as basic knowledge required for orientation in the political, social and 
economic world; problem solving capacity in everyday and critical situations; self perc^tion in the 
social context; and percq>tion of critical human vaJues. A small subgroup is developing a more 
detailed plan to identify questions and issues and design a questionnaire on national practices in a 
small number of countries* 

A second area of new work concerns outcomes that derive from the goals for education systems 
rather than for individuals. This work derives from the perspective that what is an output from one 
part of the system can be an input for another and vice versa. One of the important outcomes of 
education systems is whac opportunities are providul or offered to students. A second subgroup is 
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investigating ways in which these system ouQ)uts can be systematically defined and measured. The 
end product of this work may take the form of qualitative rather than quantitative indicators, and thus 
an important part of this work will be to investigate new ways of presenting data. 

Network A has undertaken several research and development activities that will result in indicators for 
future editions of Education at a Glance that go beyond the current subjects. These activities include: 

o A study of national assessment systems, with a view to determining a) what countries 
assess and how and b) the extent to which these national assessments could provide 
comparable information on outcomes, or, if found not to be suitable in their present 
form, how they might be adapted to meet this purpose (with particular attention being 
given to issues of anchoring and developing an item bank). The project will describe 
both traditional examination systems serving certification or selection purposes and 
assessment programs designed to monitor, evaluate, or hold accountable systems or 
individuals, with an emphasis on the latter. This work is being carried out by 
Thomas Kellaghan in Ireland. 

o The development of indicators for cross-curriculum competencies. The network plans 
to develop indicators for civic competencies and, finally additional cross-cutting 
student outcomes. This work is being carried out by the CCC subgroup headed by 
Uri Trier in Switzerland. 

o The development of information about national goals, stated and realized curriculum, 
and student performance. The initial efforts will include completion of a survey of 
national goals and a review of stated curriculum. This work is being carried by the 
GOALS subgroup headed by Marit Granheim in Norway and Sten Petterson in 
Sweden. 

o The preparation of new analyses built around subject-bound data. In particular. 

Network A is considering gender-based indicators, percent competent and age 9-14 
progression in reading literacy, mathematics, and sciences for the third edition of 
Education at a Glance. 

These efforts were in combination with work on subject-bound indicators and standards development 
for major elements of an emerging strategic plan for Network A activities. 

Network A remains cognizant of the limited nature of currentiy existing cross-national surveys of 
education outcomes, and supports efforts for OECD to take a more acUve role in defining its data 
needs. Network A is also pursuing the development of indicators of non-curriculum bound outcomes. 
A subgroup, under the leadership of Uri Trier of Switzerland has been charged with the 
developmental work in this area. The group is considering developing indicators in five areas: 
knowledge of the social and political world, problem-solving, conununication, self perception, and 
knowledge of democracy. The subgroup has undertaken a preliminary survey to establish the 
importance of non-curriculum bound outcomes areas in national goals. It has also undertaken a small- 
scale research project to identify these areas in the curriculum in four countries and commissioned 
expert psq[)ers in each of the five areas of interest. 

The second set of education indicators (appearing in the second edition of Education at a Glance, 
1993) have come at a time when OECD Member countries face serious problems of sluggish growth 
and rising unemployment. Because of this situation, in 1992 the OECD Council of Ministers 
requested that the Secretariat undertake a major study of the causes for, and possible remedies to, 
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current high levels of unemployment. The study's preliminary findings, presented in 
Employment/Unemployment Study: Interim Report by the Secretary-General (1993), include human 
resource development in the range of strategies called for to boost employment and reduce 
unemployment. The report notes in particular the need for sound initial education, better integration 
between academic and vocational studies, appropriate linkages and partnerships between schools and 
employers, workforce relevance of tertiary education, and an adult training system adapted to the 
needs of employers, workers, and non-workers. The r^rt further recognizes that the relationships 
between education, training, employment, and productivity growth are complex. The formulation of 
education policy depends on broad social, economic, and cultural factors. 

The monitoring of progress and experimentation in systems of education depends heavily on 
indicators that enable government authorities and other groups to judge the context and functioning of 
education and the results achieved. Education indicators can reveal some of the most critical 
weaknesses of education systems, and can aid the design of corrective policy. Problem areas that 
need to be addressed are the une<}ual distribution of opportunities, rigidity of student streaming and 
tracking systems, and poor use of human resources caused by ineffective education and training 
programs. The high costs of education are accompanied by discrepancies in per student expenditures. 
Data presented in Education at a Glance (1993) can contribute to identifying seme critical problems 
and establishing new priorities. 

The 1993 edition of Education at a Glance includes economic, social, and demographic context of 
education; results of education; and information on costs, resources, and processes. Thirty eight 
indicators offer a body of information on crucial aspects in education policy - investment levels, 
financing and staffing, decision-making, level of participation, student tracking, student achievement 
in key subjects and graduation rates, and whether some levels and types of education give better 
protection than others against the risks of unemployment. 

For each topic, the indicators presented in this publication reveal many similarities and dissimilarities 
across the countries, and raise questions concerning goals and efficiency of countries' varied 
educational policies. The indicators also provide important information about new developments in 
OECD countries' education systems. The primary result of the set of indicators in this edition of 
Education at a Glance affirms that education and training are part of the problem of a high level of 
unemployment as well as part of the solution. 

Education at a Glance is not a historical study but an instrument for monitoring and guiding 
educational policy. For this reason the data have to be processed quickly and published without 
undue delay. Education at a Glance should also reflect reality, which means that the changing 
conditions of education in a complex mtemational environment must be taken into account. For this 
reason all the OECD Member countries have engaged in the demanding exercise of updating 
information, which has involved substantial improvements in the data collection, processing, analysis, 
and reporting procedures. The year 1993 was devoted to the task of improving the procedures for 
data collection and transfer. The experience the INES participants have gained in producing the 
second edition of Education at a Glance is an important resource for achieving a formal and 
standardized protocol for the regular production of international education indicators. This is one of 
the major long-term objectives of the INES Project. 

The development of the OECD set of international education indicators and the publication of 
Education at a Glance are generating a series of studies at the national level which complement the 
international perspective. National indicators studies have been produced or are being developed in 
Canada, Belgium, Denmark, France, and the United States. This is a very important development 
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because the policy-relevance and the adequacy of any interpretation of the OECD indicators depends 
on whether ^e international data are supplemented by country-specific, within-system information. 

The OECD secretariat has received the following reports: The State of the School: 30 indicators of 
the educational system (Ministry of National Education and Culture, France); Geography of the 
School (Ministry of National Education and Culture, France); Facts and Figures, Education Indicators 
Denmark (Ministry of Education and Research, Denmark); The National Education Goals Report, 
1992: Building a Nation of Learners (National Education Goals Panel, U.S.A.); How in the World 
Do Students Read? Effective Schools in Reading. Implications for Educational Planning; and 
Teaching Reading Around the World (Inicniational Association for the Evaluation of Educational 
Achievement, The Netherlands). 

The Policy Review and Advisory Group were joined in their January 1993 meeting by three invited 
consultants: Erik Hanushek (United States), Claus Moser (United Kingdom), and Francois Orivel 
(France), ^ese consultants concluded that the first edition of Education at a Glance marks a major 
step in the development of international education statistics and indicators. They believe that the 
INES study serves an important function in driving improvement in data collection at the national 
level. They also identified the need for more adequately addressing the constituencies of industry, 
business, and the general public; more fully developing indicators concerning outcomes of education; 
and gathering more information about the contexts — demographic, cultural and ecological - in which 
education systems operate. In addition, they concluded that much value added would be achieved if 
time series and forecasts were included in a future edition, and that an indicator report must strike a 
fme balance between complexity and simplicity. They noted that some indicators may be too close to 
elementary statistics, whereas many others are too complex in the form in which they are shown. 

It may become necessary to publish two versions in the future: an indicator report addressed to 
specialists, and another, much shorter and simpler report intended for the general public. They also 
stated that clarification is needed on how indicators can be fruitfully employed not merely for 
informing debate but also for decision-making. 

The future of the indicators project depends on the capacity of the OECD to deal with the issue of 
data collection. In developing new indicators for the third edition of Education at a Glance, the 
Networks and the Technical Group required data not currently available on a regular basis. Another 
problem is that some of the data art^ being provided by different sources. The Policy Review and 
Advisory Group agreed that it was urgent to address the information needs of the OECD education 
statistics and indicators activity, and decided to explore ways for the OECD to play an active role in 
the gathering of education data. 

A January 21, 1993 letter from T.J. Alexander, director of the OECD Directorate for Education, 
Employment, Labour and Social Affairs, to the Education Committee of the CERI Governing Board 
informed the Members of the outcome of consultations with international organizations to improve the 
collection, analysis, and publication of CERI indicators and education statistics. OECD, UNESCO, 
and EUROSTAT have agreed on the need to adapt the ISCED classification system and to revise the 
common questionnaires in the light of new data requirements that have arisen. With the support of 
OECD and EUROSTAT, UNESCO will carry principal responsibility for the adaptation of the ISCED 
system to current needs. In consultation with EUROSTAT and UNESCO, OECD will take the lead 
in updating and revising the common questionnaires. OECD and EUROSTAT will jointly develop 
the communication technology needed for the electronic transmission of data from the capitals to the 
international organizations. 
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OECD will give priority to the improvement of the indicator templates, and OECD and EUROSTAT 
have agreed to publish jointly the INES Technical Handbook in late 1994 or early 199S. This will 
guarantee that the defmitions, calculation procedures, and technical criteria used by the organizations 
are fiilly compatible. EUROSTAT will carry out a feasibility study concerning the production of 
education indicators that are particularly suited to the needs of the European Community. 

TimetabU 

1987 The International Conference on Educational Indicators was convened in 
Washington, D.C. (November) 

1988 The Second International Conference on Educational Indicators was convened in 
Poitiers, France. (March) 

Networks were established. 

Reports of Network 2 countries were transmitted to the United States. 

Scientific Advisory Group met. 

A preliminary draft plan for indicators was prepared. 

1989 A Network seminar was convened to establish a working plan. 

The Scientific Advisory Group met to review and coordinate network plans. 

A Network analytical draft paper was completed. 

A General Assembly of five networks met in Semmering, Austria. 

A Network analytical paper was delivered to CERI. 

The Phase 2 workplan endorsed by the CERI Governing Board was approved. 
(November) 

1990 A Network A Planning Meeting was convened in Washington, D.C. (March) 
The Consultative Group met in Washington, D.C. (May) 

Network A met in Washington, D.C. (June) 
The revised Network A survey was sent to network members. (July) 
The survey was returned with members' comments. (August) 
The survey was finalized and distributed. (August) 

The Consultative Group met with network leaders in Paris, France. (September) 

The survey was carried out and completed. (October) 

The finalized criteria paper was sent to members. (October) 

The Consultative Group met in Australia. (November) 

A list of proposed indicators was sent to members. (November) 

1991 Network A met in Washington, D.C. (January) 
Network leaders met in Paris. (February) 

The Consultative Group met in Washington, D.C. (April) 

The Consultative Group met in Paris. (June) 

Network A met in Breckinridge, Colorado. (June) 

The Network A final report was completed. (August) 

The INES Project General Assembly met in Lugano, Switzerland. (September) 

1992 Education at a Glance was published (first edition). 
Networks A and D met in Paris. (March) 

The Technical Group met in Paris. (June) 
Network A met in Oslo. (June) 

The Policy Review and Advisory Group met. (September) 
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Network D met in Edinburg. (September) 
Network C met in The Hague. (November) 

1993 Education at a Glance was published (second edition). 
Making Education Count was published. 

Network A met in Vilamoura, Portugal. (February) 

INES staff met officials from EUROSTAT, (February and March) 

The Technical Group subgroup on education indicators met. (March) 

Six members of the Network A subgroup on GOALS met in Oslo, Norway. (March) 

Members of Network A CCC group met in Geneva, Switzerland. (March) 

Network B met in Paris, France. (March) 

National Co*ordinators met in Paris, France. (March) 

Network D met in Madrid, Spain. (March*April) 

The Technical Group and Networks A and C met in Paris. (June) 

The Policy Review and Advisory Group met. (July) 

The Technical Group met. (October) 

1994 Education at a Glance will be published (third edition). 

INES Technical Handbook will be published jointly by OECD and EUROSTAT. (late 
1994 or early 1995) 

1995 Several compilations of methodological contributions achieved in INES Project Phase 
3 will be published. 

1996 An international database, an on-line network for collecting and disseminating data, 
and a cost-effective procedure for processing data will be established. 
Indicators function will become part of the regular responsibility of the OECD 
Education Division. 

Publications 

Education at a Glance (first edition), 1992 (30 indicators). Education at a Glance (second 
edition), 1993. The INES Project will result in successive editions of Education at a Glance 
(containing results of indicator calculations); a handbook of international educational 
indicators (containing background information on the development of the project and 
templates for each indicator); and a set of theoretical and technical papers on various indicator 
topics. 

Making Education Count: Developing and Using International Indicators (sunmier 1993). A 
reader bringing together 18 chapters on the development and interpretation of the international 
education indicators. 

INES Technical Handbook (to be published jointly by OECD and EUROSTAT late 1994 or 
early 1995). 

Several compilations of methodological contributions achieved in INES Project Phase 3 (to be 
published in 1995). 
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By the time the INES Project is completed in 1996, the Organisation for Economic Co- 
operation and Development and its Member countries should have succeeded in setting up an 
international database containing up-to-date and relevant education statistics and indicators; a 
fiilly computerized on-line network for collecting and disseminating the data; and cost- 
effective procedures for processing the data. 

Funding Each member country has contributed resources to this project; some have provided 
substantial additional assistance through their support to the Technical Group, the four Networks, and 
several ad hoc investigative teams. The publication of Education at a Glance has been facilitated by a 
special grant to INES by the National Center for Education Sutistics in the U.S. Department of 
Education. Each member country generally carries its own cost of participating in the INES Project. 
The lead country for each network generally has responsibility for costs incurred in meetings, 
communication between members, staff support, and publications. The U.S. portion is funded by the 
National Center for Education Statistics. 

Information Sources 

International Coordinating Center: 

Centre for Educational Research and Innovation 

Organisation for Economic Co-operation and Development 

2 rue Andre-Pascal 

75775 Paris Cedex 16 FRANCE 

International Coordinator: 

Dr. Norberto Bottani, Principal Administrator INES Project 
Centre for Educational Research and Innovation-OECD 
2 rue Andre-Pascal 
75775 Paris Cedex 16 FRANCE 

telephone: 33-1-45-24-92-50 

facsimile: 33-1-45-24-90-98 

Chair, Consultative Group (Phase 2): 
Dr. Herbert Walberg 
University of Illinois 
522 N. Euclid Avenue 
Oak Park, Illinois 60302 

telephone: 312/ 996-5580 

facsimile: 312/ 996-6400 

National Coordinating Center, Network A: 
National Center for Education Statistics 
555 New Jersey Avenue 
Washington, D.C. 20208 
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National Coordinator: 
Dr. Dawn Nelson 

International Studies, Data Development Division 
National Cent^ for Education Statistics 
555 New Jersey Avenue, N.W. 
Washington, D.C. 20208 

telephone: 202/ 219-1740 

facsimile: 202/ 219-1575 

Chair, Network A: 

Dr. Eugene Owen, Chief, International Activities 
Education Assessment Division 
555 New Jersey Avenue, N.W. 
Washington, D.C. 20208 

telephone: 202/ 219-1746 

facsimile: 202/ 219-1736 

e-mail: Eugene_Owen®doed.gov 

Binkley, Marilyn R. 

1991 Results of the First International Survey of International and Intranational Educational Outcome 
Assessment Practices. Psq^r presented at American Educational Research Association annual 
meeting, Chicago. April. National Center for Education Statistics, Washington, D.C. 

Organisation for Economic Cooperation and Development/Centre for Educational Research and Innovation 

1989 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 2; No. 3 

(December). CERI and Education and Training Division of OECD, Paris, France. 
1989 OECD/CERI International Indicators Project - Network 2: Education Outcomes Network 

Report. August. 

1989 Project on International Educational Indicators (INES Project): Governing Board Review of 
Preliminary Results and Proposal for Future Work. Draft. October 7. Paris, France. 

1990 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 4 
(March). CERI and Education and Training Division of OECD, Paris, France. 

1991 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 6 
(January); No. 7 (March). CERI and Education and Training Division of OECD, Paris, 
France. 

1991 Outcomes of Education. OECD International Education Indicators. Network A: Report of 
Phase 2. September. 

1992 Education at a Glance: OECD Indicators. OECD, Paris, France. 

1992 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 9 
(January); No* 10 (May); No. 12 (^pril). CERI and Education and Training Division of 
OECD, Paris, France. 

1993 Education at a Glance: OECD Indicators. OECD, Paris, France. 

Phillips, Gary W. 

1990 OECD/CERI/INES Project * International Education Indicators • Network A: Student 
Outcomes - Phase Two Working Meeting. Bethesda, Maryland. June 25-27. National Center 
for Education Statistics, Washington, D.C. 

1991 OECD Indicators Project: Network A - Student Outcomes. Meeting Sumnuiry, Proposed 
Indicators, and AERA Presentation on Survey Results. Compiled for the Board on 
International Comparative Studies in Education. April 22. National Center for Education 
Statistics, Washington, D.C. 
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1992 OECD International Education Indicators Project. Status rqx>rt prepared for the Board on 
International Con^Murative Studies in Education. Sqptember 11. NCES, Washington* D.C. 

1993 OECD INES Project Network A. Status reports prqMred for the Board on International 
Comparative Studies in Education. January 12; May 26. NCES» Washington, D.C. 

1993 OECD INES Project. Status report prepared for the Board on International Conq>arative 
Studies in Education with Jay Moskowitz and Laura Salganik. October 19. NCES, 
Washingt(Mi, D.C. 

1994 OECD INES Project. SUtus report prqMued for the Board on International Comparative 
Studies in Education. January 11. NCES, Washington, D.C. 

1993 Technical Standards for International Indicator DaU, OECD/CERI International Education 
Indicator Project. Paper presented at American Educational Research Association annual 
meeting, Atlanta Georgia. Revised May 25 for presentation to Board on International 
Comparative Studies in Educaticm June 18. NCES, Washington, D.C. 

U.S. Department of Education 

1990 OECD/CERI/INES Project, International Education Indicators, Networic A: Student 
Outcomes. Briefing book for meeting. June. 



NOTE: This study summary was prepared on June 1, 1994 and sent to Gary Phillips (former chair of 
Network A) and to Eugene Owen (chair of Network A) at the National Center for Education Statistics 
for review. No suggestions for changes have been made. 
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INTERNATIONAL ADULT UTERACY STUDY 



Itttematiotud Or^amiation Statistics Canada 

Years of Data CoUeetwn Pilot: 1993 

Main data collection: 1994 

Purpose The International /> '"essment of Adult Literacy is a joint project of the Educational Testing 
Service and Statistics Canads^* Its basic objective is to concurrently profile the literacy skills of adults 
in a number of countries/languages* Methodologically the study represrats a fusion of educational 
testing and survey research* In order to build policy-relevant new data sets to feed the public 
discourse on emerging policy issues, the lALS is designed as a fusion of educational testing (of the 
type that goes on in the school system) with housdiold surveys* The objective is to concurrently 
administer the same measure in a number of languages and countries to profile adult literacy skills. 
Although this objective is simple, achieving it is complicated* 

lALS is based on the collective experience of Educational Testing Service and Statistics Canada with 
about 50,000 respondents* It conceptually close to the Survey of Literacy Skills Used in Daily 
Activities (LSUDA), which was fielded by Statistics Canada in 1989* The lALS will incorporate an 
international dimension to enable comparison of literacy profiles between countries* International 
objectives are to: 

o test the notion that the decoding and decision making skills embodied in the Canadian 
and U*S* assessments are stable across language groups and cultures 

o promote the conception of literacy underlying the direct assessments conducted in 
Canada and the United States 

o provide comparative data on adults and workers in countries that represent a cross- 
section of Canada's current and emerging trading partners 

o compare and contrast the literacy skill profiles for economically important sub- 
populations across countries and language groups. 

In addition, this study has objectives specific to Canada: 

o provide an updated profile of adult literacy abilities for Canada for comparison to that 
provided by the Survey of Literacy Skills Used in Daily Activities 

o provide sufficiently large numbers of visible minorities, seniors, recent graduates, and 
out of school youth to profile their skill levels 

o shed light on the relationship between performance, educational attainment, labor 
market participation and employment for those individuals found to be at Level 3. 

lALS promotes a modified model of learning* The old model was based on the concept that learning 
is centered exclusively in a formal system; a single learning event early in life (focus on youth); once 
acquired, literacy skills are viewed as a static commodity, and the adult milieu has no effect on skill 
retention or enhancement* The new model is based on die concept that self-directed instruction 
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complements the foratal system; Treated learning events (focus includes adults); slall attrition; and 
opportunity to learn plays a crucial role in skill maintenance* 

lALS incorporates a modified model of reading and yields current estimates of the magnitude of the 
literacy **problem** across a continuum of skill levels. lALS allows analysis of the covariates of 
measured performance* This information is useful in understanding etiology of the phenomenon and 
identifying populations at risk. This study model offers information on behaviors, nK)tivational 
factors, and self-assessed skill levels* These data are useful in identifying barriers to remediation and 
are key input for optimizing program design, delivery, and marketing* The lALS model also renders 
explicit the relationship between tested performance and traditional literacy proxies, such as 
educational attainment* This information is useful in projecting the evolution of skill profile and in 
generating profiles for small geogrs^hic areas. 

Some higher order benefits of the lALS are: unique collaboration b^een policy makers, collection 
experts, and measurement experts that have the subject matter knowledge; promotion of NALS 
measurement protocol; and determining the cultural limits of the conceptual framework. 

U.S. Department of Education recent interest in the current lALS design (which now contains 
connections with the work force) is in its relating to the importance placed on end results of the 
educational process and to U.S* national education goals, particularly the fifth goal on work force 
skills and adult literacy concerns (which in international comparison is work force competence). As 
Statistics Canada has assumed leadership in this project, it becomes possibly the first govemment-to- 
government international study. The U.S. Department of Education, as well as the U.S. D^artment 
of Labor and other U.S. government agencies are interested in a joint collaboration, co-funding the 
U.S* participation in this study* It is smaller in scale in terms of the number of countries that might 
be involved, more clearly targeted towards policymakers' interest, and coordinated or carried out by 
government agencies that have the same concerns about the quality of the studies demanded for 
supporting policy decision making. 

The U.S. Department of Education is particularly interested in conducting a linkage study to the U.S. 
national adult literacy survey, to link the rich background information to an international context and 
look at the performance of different groups in the U.S. population relative to other countries, as well 
as a country*to-country comparison. 

An additional higher order benefit of the lALS is getting other governments interested in govemment- 
to-govemment initiatives that would become another model for conducting international studies and 
perhaps a more efficient and effective model for doing that* In the interest of stability and viability of 
the organization structure for international studies, an lALS govemment*to-government initiative 
would provide a valuable opportunity to begin networking with stable government organizations - 
which have people who are concerned about the quality of a study and who have data management 
experience as well as the existence of a survey that can serve as tiie vehicle for a study - provides an 
interesting new approach for conducting international studies. 

OrgoniTati Qn and Managemeni An agreement in principle with National Center for Education 
Statistics has been established that the study should become a joint government undertaking (rather 
than an Educational Testing Service-Statistics Canada undertaking). Negotiation of terms is underway 
with NCES* Whatever the outcome of these negotiations, ETS will continue to play a central 
technical role in the study. 
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Participanis Canada, France, Germany, Ireland, the Netherlands, Poland, Switzerland, and 
the United States (Mexico had to withdraw). Discussions are underway with Australia, Italy, 
Js^an, New Zealand, Portugal, and the United Kingdom concerning the costs and criteria for 
joining lALS at this juncture. 

Sample The study will yield a sample of 3,000 adult Canadians. This sample will be 
enriched to ensure a minimum yield of SOO cases for seniors, in-school youth, out-of*school 
youth, official*language visible minorities, and allophone visible minorities, bringing the total 
sample to approximately 4,800. lALS methodology requires a representative probability 
sample of suiults 16-65, a minimum yield of roughly 3,000 cases, and 500-600 cases required 
for each domain/geogr^hy to be profiled. 

Procedures and Summary of Content The study will include a background questionnaire, a brief set 
of core items, and a large main test set of simulation tasks. The core items and simulation tasks will 
assess three aspects of literacy: prose literacy, document literacy, and quantitative literacy. In order 
to ensure broad coverage of the tfiree literacy domains and not to overburden individual assessment 
takers, a Balanced Incomplete Block (BIB) assessment design will be utilized. Item response theory 
(IRT) will be utilized to estimate performance on each of the three literacy domains. Results will be 
reported in terms of proficiency levels. 

lALS technical challenges include development of a multilingual test with good psychometric 
properties; ensuring comparability of key elements of a background questionnaire (e.g., educational 
attainment, industry, occupation); extension of a test to more skill dimensions (e.g., problem solving, 
communication); interviewer administered collection in each respondent's home; and interview length 
of roughly 90 minutes. (In Canada each case costs $150-$2(X).) An interviewer administers each 
element serially in a respondent's home* 

The lALS background questionnaire includes classification variables (housdiold arrangements, place 
of birth and immigration, income, education, language, and ancestral/parental information); labor 
market activity Oabor force history, labor force status, career strategies); workplace skills (general 
skills and literacy skills); training/adult education (nature of, attitudes to, outcomes of); and general 
literacy concerns. 

Data Collection and Antdvses Assessment is virtually complete, having passed the final series of 
reviews. The background questionnaire is also in the final review stage, with participants having 
agreed to a set of mandatory items to be carried in all countries. 

Canada, France, Germany, Ireland, and the Netherlands have all completed large scale pilot surveys 
that have been analyzed by ETS. Switzerland and Poland have conducted more modest tests of their 
instruments. In April 1994 participating countries met in Hamburg to discuss the results of the pilot 
surveys and to plan for implementation of the main study (main data collection in September 1994). 

Decisions made at the April 1994 meeting were: 

0 Based on an analysis of pilot assessment data by ETS and Stan Jones, Canada's 

technical consultant on the study, participating countries have agreed to a final item 
selection and test design that will afibrd the psychometric equivalence needed to 
support international comparisons. 
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0 Countries have committed to field a comprehensive set of standardized background 
variables including modules on adult education participation and income. Both of 
these modules had been problematic to at least some of the countries prior to piloting. 

0 Countries have agreed to a proposed content and structure for the international 

comparative vepon^ as well as a process for vetting successive drafts of the report as 
they become avaU2d)le* Approval in principle was also reached to have the report 
rdeased as a joint Statistics Canada-OECD publication* 

0 A compromise was reached concerning access to microdata following release of the 
international comparative r^rt. The Canadian tradition would have seen non- 
confidential microdata being released concurrently with the comparative tepott. 
Countries have agreed in principle to a controlled release for a period of a year after 
which the dataset would become freely available. 

Timetable 

1993 The survey was piloted. 

1994 Main data will be collected. (September) 

Publications 

International comparative report: A single international report will focus on the similarities 
and differences in the literacy profiles of participating countries and their implications for 
national and international policy making. This report will be released as a joint Statistics 
Canada-OECD publication. 

Canadian rq>orts will include a first report to present a comparison of the lALS data to 
LSUDA data; a second report that will focus on rare population subgroups for which LSUDA 
failed to provide sufficient sample; a third report that will reflect on the policy implications of 
the comparisons to other participating countries. 

Database 

Name International Adult Literacy Survey (lALS) 

Description A public-use microdata file and associated documentation in electronic form 
containing results from the International Adult Literacy Survey (lALS), a nine country direct 
assessment of prose literacy, document literacy^ and numeracy. 

To whom available All researchers. 

Restrictions Restricted access to microdata file January-December 1996. Interested users will 
be required to submit a research proposal, which will be vetted by a review committee. Data 
from 1997 file will be freely accessible. 

gy what means available The file will take the form of a CD-ROM, which will contain the 
data file, extraction software, and related documentation. 

Charfe to user To be determined. 
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When availabU January 1996. 



For further Information about access to this database^ contact: 
Jean Pignal 

The Special Surveys Division, Statistics Canada 
Ottawa, Ontario, Canada KIA 0T6 

telephone: , 613/ 951-3317 

facsimile: 613/ 951-0562 

electronic mail: pignjea®statcan.ca 

Funding The European Community and lALS European participants have agreed to underwrite 
$200,000 of the estimated $400,000 cost of international overheads associated with the study. The 
National Center for Education Statistics has also agreed in principle to support the study and continue 
to search for the funds necessary to do so. The bulk of this money will be directed towards the 
purchase of services from the Educational Testing Service* 

The Educational Testing Service has provided a proposal to Statistics Canada to cover their continued 
involvement in the study, and Statistics Canada, as study manager, has agreed to fund most of what 
has been presented. In addition. Statistics Canada has been working with the U.S. Bureau of the 
Census to develop a collection strategy for the U.S. linking study, which the National Center for 
Education Statistics can fund. This study, which would see the administration of a hybrid 
NALS/IALS assessment to 3,150 Americans, is a critical element in providing the statistical basis for 
placing other countries on the U.S. scales. NCES has also agreed in principle to sign an inter-agency 
agreement to cover this work. 

Information Sources 

T. Scott Murray, Assistant Director 
Household Surveys, Statistics Canada 
Ottawa, Canada KIA 0T6 

telephone: 613/ 951-9476 

facsimile: 613/ 951-0562 

Internet: scotmur@statcan.ca 



Statistics Canada, The Special Surveys Group 

1992 An International Assessment of Adult Literacy: A Proposal. Draft. December. 

Murray, T. Scott 

1993 Update of 1992 draft proposal. May 28 letter to Dorothy M. Gilford, director. Board on 
International Comparative Studies in Education. 

1993 International /.dult Literacy Survey. Status report to the Board on International Comparative 
Studies in Education. October. 

1994 International Adult Literacy Survey. Status reports to the Board on International Comparative 
Studies in Education. January and April. Presentation to the board. May. 



****** 
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NOTE: This study summary was reviewed and edited by T. Scott Murray at Statistics Canada in 
Ottawa on May 31, 1994; database information was provided by Murray on December 9, 1994. 
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INTERNATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 



Internation al Organization Educational Testing Sen ice 

Years of Data CoUection 

International Assessment of Educational Progress * I: 1988 
International Assessment of Educational Progress - II: 1991 

Purpose 

Since 1983 Educational Testing Service has administered the National Assessment of 
Educational Progress and related projects. NAEP is an ongoing, congressionally mandated 
project established to conduct national surveys of the educational attainments of students in the 
United States. Its primary goal is to determine and report the status of and trends over time 
in educational achievement. NAEP was initiated in 1969 to obtain comprehensive and 
dependable national educational-achievement data in a uniform, scientific manner. 

Iriternational Assessment of Educational Progress I: The purpose of lAEP-I was to 
collect and report data on what students Icnow and can do, on the educational and cultural 
factors associated with achievement, and on students' attitudes. IAEP*I assessed mathematics 
and science achievement of 13-yearH)ld students in the United States and five other countries. 
The United States joined with the five other countries to explore the feasibility of reducing the 
time and money requirements for international comparative studies by capitalizing on the 
National Assessment of Educational Progress materials and procedures. lAEP-I permitted 
interested countries to experiment with NAEP technologies to determine their appropriateness 
for local evaluation projects. 

International Evaluation of Educational Progress - II: The purpose of lAEP-II was to 
collect and report data on what students know and can do, on the educational and cultural 
factors associated with achievement, and on students' attitudes. lAEP-II assessed mathematics 
and science skills of samples of 9- and 13-year-old students from the United States and 19 
other countries, using technology developed for the National Assessment of Educational 
Progress. By utilizing existing NAEP technology and procedures, time and money required 
to conduct these international comparative studies was reduced and many interest^ countries 
were able to experiment with the innovative psychometric techniques. 

Countries that participated are large and small, rich and poor, and have varied ethnic, 
religious, language, and cultural traditions. Educational goals, expectations, and the meaning 
of achievement vary among each of these countries. Each of the countries that participated 
did so for its own reasons: to compare its results with those of neighbors or competitors; to 
learn about the educational policies and practices of countries whose students seem to 
regularly achieve success in mathematics and science; to establish a baseline of data within its 
own country against which progress could be measured in the future. 

All participants shared a common interest in identifying what is possible for 9- and 13-year- 
olds to know and to be able to do in mathematics and science. Knowledge of what is possible 
produces new enthusiasm, raises sights, establishes new challenges, and ultimately can 
improve personal and societal performance. The assessment information can be used by each 
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country to set goals and develop curriculums and practices in harmony with its own values 
and culture. 

An optional geography component was included in this project, as a first step toward 
acquiring international measures of geography skills, knowledge, and education. Through this 
optional geography component, participants hoped to identify behaviors and practices that 
contribute to high levels of achievement, so that education policy makers would have 
information that might enable them to improve domestic performance in concert with national 
educational goals. 

Organization and Mnnm^ntnt 

International Assessment of Educational Progress - 1: A project director, located at the 
Educational Testing Service, worked in collaboration with the National Foundation for 
Educational Research (UK); Ministry of Education and Science (Spain); New Brunswick, 
Ontario, and Quebec Ministries of Education (Canada); Laval University data analysts; and 
Educational Testing Service consultants. 

International Assessment of Educational Progress - II: International coordination was done 
by staff of the Canadian Data Analysis Group, Educational Testing Service, and Westat. In 
all countries except Brazil and Mozambique an independent, trained observer interviewed the 
country project manager about all aspects of the project and visited one or more test 
administration sites. In most cases, the observer was fluent in the language of the assessment. 

Decisions concerning the design and implementation of the project were made collaboratively 
by the representatives of the provinces and countries involved in the survey. The National 
Academy of Sciences/National Research Council's Board on International Comparative Studies 
in Education reviewed plans for the project at several stages of its development and made 
suggestions to improve the technical quality of the study. 

Design 

Participants 

International Assessment of Educational Progress - 1: Five countries and four Canadian 
provinces with extensive experience in large-scale assessment participated. (Canada does not 
have a federal system of education.) Twelve student populations were included from: British 
Columbia, Ireland, Korea, New Brunswick (English), New Brunswick (French), Ontario 
(English), Ontario (French), Quebec (English), Quebec (French), Spain, United Kingdom 
(sample drawn from students in England, Scotland, and Wales), and United States. Students 
were 13 years old (bom January 1, 1974-December 31, 1974), and were selected from public 
and private elementary, middle, and secondary schools. 

International Assessment of Educational Progress - II: Twenty countries assessed the 
achievement of 13-year-oId students and 14 assessed 9-year-old students in mathematics and 
science. In some cases, participants assessed virtually all age-eligible children in their 
countries; in other cases they confined samples to certain geographic regions, language 
groups, or grade levels. In some countries significant proportions of age-eligible children 
were not represented because they did not attend school. Low rates of school or student 
participation in some countries mean results may be biased. 
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Country participation is as follows: 



Brazil cities of Sao Paulo and Fortaleza, restricted grades, in-school population 

Canada four provinces at age 9 and nine out of 10 provinces at age 13 

China 20 out of 29 provinces and independent cities, restricted grades, in-school population 

England all students, low participation at ages 9 and 13 

France all students 

Hungary all students 

Ireland all students 

Israel Hebrew-speaking schools 

Italy province of Emilia-Romagna, low participation at age 9 

Jordan all students 

Korea all students 

Mozambique cities of Maputo and Beira, in-school population, low participation (Mozambique did 

not assess Science.) 

Portugal restricted grades, in*school population at age 13 

Scotland all students, low participation at age 9 

Slovenia all students 

Soviet Union 14 out of IS republics, Russian-speaking schools 

Spain all regions except Cataluna, Spanish-speaking schools 
Switzerland IS out of 26 cantons 

Taiwan all students 

United States all students 



The geography portion of the lAEP was a special probe into the geographic knowledge and 
skills of 13*year-oIds. Twenty countries assessed the mathematics and science achievement of 
13-year-oIds; of these 20 countries, nine countries also administered the optional geography 
component: 



Canada eight out of 10 provinces 

Hungary all students 

Ireland all students 

Korea all students 

Scotland all students 

Slovenia all students 

Soviet Union 14 out of IS republics, Riissian-language schools 

Spain all regions except Cataluna, Spanish-language schools 

United States all students 



Sample 

International Assessment of Educational Progress - 1: A random sample of about 2,000 
students from 100 different schools was selected from each populatior.. In the United States, 
the sample size was about 1,000 students in 200 schools. A total of 24,000 students was 
assessed. 

International Assessment of Educational Progress - II: Typically^ a random sample of 
3,300 students from about 110 different schools was selected from each population at each age 
level; half were assessed in mathematics and half in science. A total of about 175,000 9- and 
13-year-oIds (born in calendar years 1981 and 1977) were tested in 13 different languages in 
March 1991. Some countries drew samples from virtually all children in the appropriate age 
group; others assessed only children in specific geographic areas, language groups, or grade 
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levels. The definition of populations often followed the structure of school systems, political 
divisions, and cultural distinctions. 

Procedures and Summan o f Content 

International Assessment of Educational Progress - 1: All countries and provinces followed 
standardized administration procedures and administered the assessments during February 
1988. 

International Assessment of Educational Progress - II: Participants recognized fundamental 
differences from country to country, but assembled tests that focus on common curriculum 
elements. In order to form contexts for interpreting student achievement data, they added 
questions about students' home background and classroom experiences and the characteristics 
of the schools they attend. As a survey research project, this assessment could not establish 
cause-and-effect relationships, but it could provide clues that might help to explain high and 
low performance. 

The lAEP-II assessment used a four-part survey: a main assessment of 13*year-oIds' 
performance in mathematics and science; an assessment of 9-year-olds' performance in 
mathematics and science; an experimental, performance-based assessment of 13-year-olds' 
ability to use equipment and materials to solve mathematics and science problems; and a short 
probe of the geography skills and knowledge of 13-year-oIds. All countries participated in the 
main assessment of 13-year-oIds; participation in the other assessment components was 
optional. 

Each participating country was responsible for carrying out all aspects of the project, 
including sampling, survey administration, quality control, and data entry using standardized 
procedures that were developed for the project. Several training manuals were developed for 
the project. Several international training sessions were held, during which each step of the 
assessment process was explained. 

The assessment was developed through a consensus-building process that involved curriculum 
and measurement experts from each of the participating countries and provinces. Several 
existing NAEP frameworks and the lAEP-I framework were reviewed by participants and 
evaluated as to their appropriateness for their own countries' curriculums. Together, the 
participants then adapted the NAEP frameworks to reflect an international consensus of 
subject-specific topics and cognitive processes that they believed reasonably reflected 
curriculums being implemented in their own school systems. 

The nine countries that participated in the geogr^hy component participated in the 
development of the geography framework that guided the design of the instruments. 
Curricular experts in each country reviewed the appropriateness of all potential questions for 
their own students. 

Once the participants had agreed upon common frameworks and the relative emphases that 
would be placed on each topic and cognitive process category of the assessment, more than 
one-half submitted test items from their countries' own assessment programs that they felt 
were appropriate and met the requirements of the lAEP assessment. Many questions from the 
United States' NAEP assessments were included as well. The items with the highest ratings 
across all countries were placed into a pool of acceptable questions from which a subset was 
selected and pilot-tested in all of the participating provinces and countries. 
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While acceptable to all, the content areas measured in the tests were not equally representative 
of each country's curriculum and the multiple-choice question testing format was not equally 
familiar to students from all countries. Countries differ in the age at which students start 
school and in policies for promotion, students at age 13 were further along in their schooling 
in some countries than in others. 

Data Collection and Amdvses 

International Assessment of Educational Progress - I: Students were administered a 45- 
minute mathematics assessment (63 questions) and a 4S-minute science assessment (60 
questions), selected from the total pool of 281 mathematics and 188 science questions used in 
the 1986 Unites States' National Assessment of Educational Progress. In addition, students 
answered questions about their school experiences and attitudes, and teachers rated students' 
exposure to the concepts tested by the items. The assessment was administered in February 
1988, except in the United States, where the data were collected during the January-through- 
middle-March NAEP assessment. 

Each country and province was responsible for developing a data file following a standard 
format, for checking ranges of responses, and resolving inconsistencies in the data. Quebec 
Ministry of Education staff also checked the fdes. ETS staff calculated weights for the United 
States and Canadian participants and verified weights for other participants. A Laval 
University (Quebec) research team conducted data analysis in consultation with ETS 
researchers and data analysts. 

The first stage of analysis involved calculation of the percentage of correct answers and 
standard errors for individual questions and groups of questions. The second stage of analysis 
involved scaling of mathematics and science results using item response theory technology. 
The assessment collected opportunity-to-leam information to determine to what extent students 
in the participating populations had been exposed to various mathematics and science content 
areas covered in the lAEP questions. 

Educational Testing Service measurement specialists worked with colleagues from five other 
countries to translate and adapt techniques used in the United States for the National 
Assessment of Educational Progress. Achievement results that permit comparisons and 
present valid and reliable findings were presented in A World of Differences: An 
International Assessment of Mathematics and Science. Each participating country had the 
opportunity to experiment with new measurement practices. Cost-effectiveness of sampling 
techniques, the power and limitations of Item Response Theory, and the usefulness of new 
reporting techniques were demonstrated. Experts in each participating country had hands-on 
experiences with problems and potential of new assessment techniques. 

International Assessment of Educational Progress - II: The achievement tests lasted one 
hour. The tests given to 9-year-olds included 62 questions in mathematics and 60 questions in 
science. The tests given to 13-year-olds included 76 questions in mathematics and 72 
questions in science. In addition, students at each age spent about 10 minutes responding to 
questions about their backgrounds and home and school experiences. Finally, students in 
countries assessing geography spent an additional 10 minutes responding to questions on 
geography. School administrators completed a school questionnaire. 

National Center for Education Statistics commissioner Emerson Elliott emphasized factors to 
be kept in mind in interpreting the results of this assessment: 1) the importance of cultural 
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differences and of each country's educational traditions and practices and 2) the consensus 
process that produced the mathematics and science frameworks used in the development of 
lAEP II reflects only the curricular elements common to all participating countries. 

Data analysis carried out between S^tember and December 1991 involved checking 
dimensionality of the mathematics and science items at each age level through factor analysis 
techniques and item calibration on each of the dimensions established; computation of item 
and average percents correct; ability scoring using the plausible-values technology developed 
for the NAEP study; scale anchoring; and linkage of the two age groups. 

Assessment results were obtained for 52 populations: 







9-year-olds 


9-year'olds 


13-year-olds 


13-year-oIds 


PopuUtion 


Language 


mathematics 


science 


mathematics 


science 


Brazil 


Portuguese 


- 


- 


X 


X 


Canada 












Alberta 


English 


- 




X 


X 


Br.Col. 


English 


X 


X 


X 


X 


Manitoba 


English 


- 


- 


X 


X 


Manitoba 


French 


- 


- 


X 


X 


NewBr. 


English 


X 


X 


X 


X 


New Br. 


French 




- 


X 


X 


Newfound. 


English 


- 


- 


X 


X 


Nova Sc. 


English 


- 


- 


X 


X 


Ontario 


English 


X 


X 


X 


X 


Ontario 


French 


X 


X 


X 


X 


Quebec 


English 


X 


X 


X 


X 


Quebec 


French 


X 


X 


X 


X 


Satkatch. 


English 


— 


— 


X 


X 


Saikatch. 


French 






X 


X 


China 


Mandarin 






X 


X 


England 


English 


X 


X 


X 


X 


France 


French 






X 


X 


Hungary 


Hungarian 


X 


X 


X 


X 


Ireland 


English 


X 


X 


X 


X 


Israel 


Hebrew 


X 


X 


X 


X 


luly 


lUlian 


X 


X 


X 


X 


Jordan 


Arabic 






X 


X 


Korea 


Korean 


X 


X 


X 


X 


Mozambique 


Portuguese 








X 


Portugal 


Portuguese 


X 


X 


X 


X 


Scotland 


English 


X 


X 


X 


X 


Slovenia 


Slovene 


X 


X 


X 


X 


Soviet Un. 


Russian 


X 


X 


X 


X 


Spain 


Spanish 


X 


X 


X 


X 


Switzerland 


French/ 

German/ 

Italian 






X 


X 


Taiwan 


Mandarin 


X 


X 


X 


X 


United St. 


English 


X 


X 


X 


X 
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Population 



13-ycar-oldt 
geography option 



13-year-K>ldi 
performance option 



Canada 
Alberta 

Britiih Columbia 

Manitoba 

New Brunswick 

Newfoundland 

Nova Scotia 

Ontario 

Quebec 

Saskatchewan 
England 
Hungary 
Ireland 
Korea 
Scotland 
Slovenia 
Soviet Union 
Spain 
Taiwan 
United States 



X 
X 
X 
X 
X 



X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 



X 



X 
X 



X 
X 



X 
X 



X 
X 
X 



X 



Analysis of the second International Assessment of Educational Progress using item response 
theory technology was completed and almanacs of results were produced in 1992. A second 
volume of the lAEP Technical Report, which describes the steps in applying item response 
theory methodology, was also produced in 1992. This technical report describes the creation 
of a reference population that was used for all subsequent analyses. After ensuring that the 
reference population adequately represented each of tiie participating populations and met 
assumptions of unidimensionality, these data were used to calibrate item perimeters. 
Proficiency scores were then estimated, making use of plausible values technology. Each age 
group was scaled separately and then linked using the Stocking-Lord procedure. The 
resulting scale was assigned a mean of 500 and a standard deviation of 100. Average 
proficiency scores were then calculated for participating populations, following the same 
procedures used for the percent correct analyses. 

Secondary analysis of lAEP results. The Educational Testing Service Center for the 
Assessment of Educational Progress prepared two short reports that focus on implications of 
the lAEP results for U.S. efforts at standard setting and curriculum reform. These reports, 
written in collaboration with U.S. mathematics and science educators, make use of item-by- 
item results and comment on what students in the United States know and can do wdl and 
where U.S. students fare behind student peers in other countries. They will be published by 
the National Science Foundation. 
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CO 



1987-1989 

Project implementation was carried out through a series of meetings to select 
assessment items, review pilot-test results, and review and interpret final results. 
Decisions were made collaboratively; follow-up coordination was provided by ETS 
staff. 

Interaational Assessment of Educational Progress - II: 

1989 Pilot testing was conducted November 1989-February 1990. 

1990 The pilot review meeting was held in Montreal. (May) 
Data was collected in the southern hemisphere. (September) 

1991 Data was collected in the northern hemisphere. (March) 

1992 Reports were published on results of lAEP geography, mathematics and science. 
An almanac of results of lAEP using item response theory published. 

Two volumes of the lAEP Technical Report were published. 

Publications 

A World of Differences: An International Assessment of Mathematics and Science. Report 
of the International Assessment of Educational Progress. Educational Testing Service. (1989) 

A World of Differences: Technical Report. Rq>ort of the International Assessment of 
Educational Progress. Educational Testing Service. (1989) 

Learning Mathematics. Report of The International Assessment of Educational Progress. 
Educational Testing Service. (1992) 

Learning Science. Report of The International Assessment of Educational Progress. 
Educational Testing Service. (1992) 

Learning Geography. Report of the International Assessment of Educational Progress. 
Educational Testing Service. (1992) 

Performance Assessment: An International Experiment. Report of the International 
Assessment of Educational Progress. Educational Testing Service. (1992) 

lAEP Technical R^rt. Report of the International Assessment of Educational Progress. 
Educational Testing Service. (1992) 

lAEP Technical Report: Volume Two. Report of the International Assessment of 
Educational Progress. Educational Testing Service. (1992) 

Two short reports on implications of lAEP results for U.S. efforts at standard setting and 
curriculum reform will be published by the National Science Foundation. 

Euadiaz 

International Assessment of Educational Progress - 1: Funding was provided by the U.S. 
Dq;)artment of Education and the National Science Foundation for overall coordination, 
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sampling, U.S. data analysis, and reporting. Participating countries and provinces acquired 
support for local data collection and coordination. 

International Assessment of Educational Progress - II: Funding was provided by The 
National Science Foundation and the U.S. Department of Education's National Center for 
Education Statistics provided a grant for the expenses of overall coordination, sampling, data 
analysis, and reporting. The Carnegie Corporation provided additional funds to cover Ae 
travel expenses of some of the participants who could not meet the financial burdens of 
traveling to the project's coordination and training meetings, held in Canada, England, 
France, Hong Kong, and the United States* Participating countries acquired support for local 
data collection and coordination. 

Information Sources 

Nancy A. Mead, International Coordinator/Project Director 
Center for the Assessment of Educational Progress 
Educational Testing Service 
P.O. Box 6710 

Princeton, New Jersey 08541-6710 
telephone: 609/ 734-1526 
e*mail: nmead®rosedale.org 



Educan Inc. 

1991 lAEP Notes. 4:September. Educan Inc., Longueuil, Quebec, Canada. 

Educational Testing Service 

1991 The 1991 lAEP Assessment: Objectives for Mathematics, Science, and Geography. 
Educational Testmg Service, Princeton, New Jersey. 

1992 lAEP Technical Report* Rq;x>rt of the International Assessment of Educational Progress. 
ETS, Princeton, New Jersey* 

1992 lAEP Technical Report: Volume Two* Report of the International Assessment of Educational 
Progress. ETS, Princeton, New Jersey. 

Lapointe, Archie E* 

1988 lAEP preliminary findings; comparison with lEA findings. Presentation to the Board on 

International Conqparative Studies in Education* November 4. 
1991 lAEP analysis and report plans. Presentation to the Board on International Comparative 

Studies in Education* January 28. 

1991 lAEP report draft* Presentation to the Board on International Comparative Studies in 
Education. September 23. 

Lapointe, Archie E*, Nancy A. Mead, and Eugene Johnson 

1990 lAEP sUtus. Presentation to the Board on International Comparative Studies in Education. 
February 2. 

Lapointe, Archie E*, Nancy A* Mead, and Janice M. Askew 

1992 Learning Mathematics* Report of the International Assessment of Educational Progress. 
Report No* 22-CAEP-Ol* Febniary. Educational Testing Service, Princeton, New Jersey. 

Lapointe, Archie E*, Janice M* Askew* and Nancy A* Mead 

1992 Learning Science* Report of the International Assessment of Educational Progress. Report 
No* 22-CAEP^* February* Educational Testing Service, Princeton, New Jersey* 
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Lapointe, Archie E., Nancy A. Mead, and Gary W. Phillips 

1989 A World of Differences: An International Assessment of Mathematics and Science. Report of 
the First International Assessment of Educational Progress. Report No. 19-CAEP-Ol. 
January. Educational Testing Service, Princeton, New Jersey. 

Lazer, Stephen 

1992 Learning About the World. Report of The International Assessment of Educational Progress. 
Report No. 22-CAEP-OS. June. Educational Testing Service, Princeton, New Jersey. 

Mead, Nancy A. 

1990 Second International Assessment of Educational Progress. Status report prepared for the 
Board on International Comparative Studies in Education. April 2S. 

1992 Second International Assessment of Educational Progress. Sutus report prepared for the 
Board on International Comparative Studies in Education. September. 

1993 Secondary Analysis of lAEP Results. Status report prepared for the Board on International 
Conq>arative Studies in Education. January. 

Semple, Brian McLean 

1992 Performance Assessment: An International Experiment. Report of the International 

Assessment of Educational Achievement. Report No. 22''CAEP-06. July. Educational 
Testing Service, Princeton, New Jersey. 



NOTE: This study summary was reviewed and edited by Nancy Mead at the Educational Testing 
Service on June 10, 1994. 
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INTERNATIONAL COMPARATIVE STUDY OF MATHEMATICS AND SCIENCE TESTS 



International Orfanizatior> The National Center for Improving Science Education 
Years of Daia CoUeetion 1993 

Purpose The establishment of U.S. national goals, the push toward better means of assessment, and 
efforts to set standards for school science have led to a great interest in defining world*class standards 
in mathematics and science achievement of precoUege students. One persistent question is the 
achievement expectations for students preparing for university admission in science or mathematics 
and/or planning to major in these fields. Therefore, the National Center for Improving Science 
Education will analyze the current mathematics and science tests given to students seeking university 
admission in several industrialized countries. 

The aim is to develop more knowledge about how mathematics and science tests intended for the most 
accomplished students differ across countries, what can be inferred from the analyses of such tests 
about learning standards set for students in these countries, and how U.S. mathematics and science 
tests can be improved to match the standards of other countries. The National Center staff believe 
that their approach to this work not only will inform U.S. policy but will contribute to international 
understanding of educational expectations embodied in tests and the place of testing in national 
education policy. 

Organization and Management The project is guided by an advisory panel that reviews drafts of the 
chapters and of the complete report and helps to refme dissemination plans. The study is taking place 
over a 28-month period (September 1992 - January 1995), during which an advisory panel was 
formed and experts contacted. Other activities include establishing a structure for analysis and 
reporting; selecting, collecting, and translating tests; analyzing tests and writing findings in a final 
report; and implementing a dissemination plan. 

Design 

Participants Advance-level examinations in mathematics and science (biology, chemistry, 
physics) have been obtained from France, Germany, Great Britain, Israel, Japan, Sweden, and 
the United States. 

Sample The populations for tests selected and analyzed closely match Population 3b, the 
'^mathematics and/or science specialist" population defined for TIMSS. 

Procedures and Summary o f Content The scope of work of the International Comparative Study of 
Mathematics and Science Tests complements the work of the lEA Third International Mathematics 
and Science Study (TIMSS). The science and mathematics frameworks developed for TIMSS were 
augmented and used to ansdyze the content and performance expectations of the selected tests. To the 
extent possible, TIMSS participant survey questionnaires will be used to gather data on what sorts of 
students are taking the tests to be analyzed, how the tests are used in each of the countries, and other 
contextual factors. The test analysis project is being carried out in conjunction with TIMSS 
leadership; there is an overlap in personnel; and all countries selected to participate in this study are 
participating in TIMSS. 

Data Collection and Analyses The International Comparative Study of Mathematics and Science 
Tests is working closely with the TIMSS International Coordinating Center to collect the tests. 
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University of British Columbia graduate students are translating tests that were not provided in 
English. 

Examinations collected for analysis are in the fields of mathematics, physics, chemistry, and biology. 
For each country, each field, and pertinent type of test, the two latest years of tests available will be 
collected, in order to have an adequate representation of the domain of knowledge and performance 
expected of students. Expected responses or response protocols and scoring rubrics is included. 



Exams Included in the National Center for Improving Science Education Analysis 


Country 


Years 


Type 


Regions 


France 


1991, 1992 


Baccalaurfeat 


Paris; Aix 


Germany 


1991, 1992 


Abitur 


Bavaria; 








Baden-Wurttenberg 


Israel 


1992 


Matriculation 




Japan 


1991, 1992 


University 


Tokyo University 






Entrance 




Sweden 


1991, 1992 


National Exam 




United Kingdom 


1991, 1992 


A-level 


Associated Examining Board; 








University of London 








Examinations and Assessment 








Council 


United SUtes 


varied, 1993 


Advanced 








Placement 





In 1994 an international team of researchers assembled by the National Center for Improving Science 
Education analyzed national examinations in mathematics and science (biology, chemistry, and 
physics). Research team members and study advisors, drawn from the United States and other 
countries, included eminent experts in science and mathematics education. The exams compared were 
endH)f-secondary school tests, given to college-bound science and mathematics students, from seven 
countries. 

Analysis of the tests in mathematics and in each of the science fields, used the TIMSS mathematics 
and science curriculum frameworks (mathematics, John Dossey; biology, Pinchas Tamir; physics, 
Kjell Gisselberg; chemistry, Dwaine Eubanks). Each of these experts will produce a chapter on the 
results of the test analyses across countries together with an overview and conclusions regarding the 
achievement and performance expectations for advanced students in the specific subject. A sununary 
chapter will be. added synthesizing the separate reports on each of the fields and providing overall 
conclusions. Analyses and draft chapters will be reviewed by prominent senior researchers (Paul 
Black, physics; Angelo Collins, biology; Dorothy Gabel, chemistry; Curtis McKnight, mathematics; 
and Jack Schwille, comparative education). Two major components of the study are item-by-item 
categorizations of the tests' (1) topics and (2) expectations for student performance. To inventory 
these two characteristics, researchers developed supplemented versions of the Science and 
Mathematics Curriculum Frameworks from the Third International Mathematics and Science Study 
(TIMSS). Consequently, researchers had between SO- 75 subject matter categories available (per 
subject), nested in three hierarchical levels, to describe an item's topic. 

Researchers used multiple categories to describe an item's topic as needed. All researchers also used 
a second dimension of the TIMSS curriculum frameworks to describe each item: expectations for 
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student performance. Twenty categories were availabie, including, e.g., understanding of simple 
information, developing explanations using scientific principles, and solving quantitative problems. 

Many other features of the exams were compared. Nine categories were used to document the 
formats of questions, e.g*, multiple choice questions (3, 4, or S response options), open*ended (types 
of short-answer questions, types of extended-answer questions), and practical tasks* Researchers also 
inspected the wording of items (e.g*, whether questions were stated negatively or positively) and the 
demands of problem-solving questions (e.g., whether questions merely required set-ups for solution or 
completely worked solutions.) Finally, more global exam features were noted such as length of 
exams and kinds of options included (or not) for students to select questions. 

In late 1994 the report on detailed comparative analysis of exams will be published. Researchers 
compared three main characteristics of the tests: topics covered, expectations for student 
performance, and types of questions. The Center's international team also compared the overall 
lengths and components of the tests, the amount of choice students have in seleaing questions, the 
complexity of the topics covered, and wording devices used in constructing questions* There is only 
one exam from each year in the report, which is too few exams for determining the range of topics 
that national exams typically assess. A full range of topics would require inspection of several years 
of exams; however the Center's analysis of exams from two years will go a long way toward showing 
the range of topics included in the exams of other countries. 

Exam topics in the Center's report are described using detailed categories that the Center helped to 
develop for a the Third International Mathematics and Science Study (a large-scale international study 
involving a large number of countries). The topic of each exam question is noted using three levels 
of increasingly detailed topic categories; then each exam's topics are summarized, making 
comparisons among exam topics possible. The rq)ort addresses what kinds of processes students 
must use to answer exam questions: whether students must give information by rote, or solve 
quantitative problems, or give explanations based on scientific principles. Categories known as 
**student performance expectations,"* which the Center helped develop for the TIMSS study, were used 
to describe student performance* Five main categories (understanding, solving problems, using tools 
and procedures, investigating and conmiunicating) and IS sub-categories were employed* The report 
extensively discusses difficulty of exams and considers many factors that should be brought to bear 
when making comparisons between the ^^difTiculty** of different countries' exams. 

In May 1994, the National Center and the American Federation of Teachers copublished What 
College-Bound Students Abroad Are Expected to Know About Biology. The volume contains 
university entrance exams from England and Wales, France, Germany, and Japan, and a U.S. 
Advanced Placement biology exam, as well as comparative desciiptions of those nations' examination 
systems* Release of this report drew considerable media attention. It is recommended by the 
National Science Teachers Association and the National Association of Biology Teachers. 

Timetabie 

1993 Exams were collected. 

1994 National examinations in mathematics and science were analyzed. 
A report of comparative examination descriptions was published. 
A monograph will be published* (late 1994) 

1995 A final report will be published. 
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Publications 

What College-Bound Students Abroad Are Expected to Know About Biology. National 
Center for Improving Science Education and National Federation of Teachers. (May 1994) 
(Similar publications on chemistry and physics are proposed.) 

The end product will be a monograph released late in 1994 that should be of considerable 
interest to policy makers and scholars concerned with standards in mathematics and science 
and with testing* A number of diverse strategies will be used to disseminate the results of this 
study as widely as possible. 

Funding Funding is provided by the National Science Foundation. 

In formation Sources 

Senta A. Raizen, Director 

The National Center for Improving Science Education 
2000 L Street, N.W., Suite 603 
Washington, D.C. 20036 

telephone: 202/ 467-0652 

facsimile: 202/ 467-0659 

e-mail: sentar@gwuvm.gwu.edu 

Ted Britton, Project Director 

The National Center for Improving Science Education 

2000 L Street, N.W., Suite 603 

Washington, D.C. 20036 

telephone: 202/ 467-0652 

facsimile: 202/ 467-0659 

e-mail: tbritton@gwuvm.gwu.edu 



National Center for Improving Science Education 

1992 International Comparative Study of Mathematics and Science Tests: Project Summary. 
1994 Content Analysis of National Science and Mathematics Examinations from Seven Countries: 
Proposal Summary. Symposium Proposal for American Educational Research Association, 
1994. 

National Center for Inq>roving Science Education and National Federation of Teachers 

1994 What College-Bound Students Abroad Arc Expected to Know About Biology. Defining World 

Class Standards Volume 1. National Center for Improving Science Education and National 

Federation of Teachers^ Washington* D.C. 
Raizen, Seota 

1992 Content Analysis of National Science and Mathematics Examinations from Seven Countries. 
Presentation to Board on International Con^wrative Studies in Education. October. 



NOTE: This study sununary was reviewed and edited by Ted Britton at the National Center for 
Improving Science Education in Washington, D.C. on May 31, 1994. 
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INTERNATIONAL STUDY OF TEACHERS' ATTITUDES TOWARD REFORM AND TEACHER 
PREPARATION FOR IMPLEMENTING REFORM 



Internatioti af OrgantTiltiifn George Washington University Institute for Curriculum Standards and 

Technology 

Purpose This study will be a cross national study of teachers' perceptions of critical issues facing 
schools^ major reforms being implemented to improve the quality of education, and the conditions 
necessary for the successful implementation of the reforms. The purpose of the study is to ascertain 
teachers' views of education refonns bemg implemented in their country. The study also will be 
designed to ask teachers what their needs are in terms of training and professional development 
experiences to implement the reforms. 

Other recent studies about education reform initiatives have focused on the viewpoints of policy 
makers. This study is unique in that it will directly involve classroom teachers in assessing the 
impact of education reforms on classrooms. 

The study will first determine whether education reform is occurring in the Washington, D.C.- 
Virginia-Maryland metropolitan area and whether teachers' professional development programs have 
been (re)designed to support the reform efforis; and then conduct an international study of teachers' 
attitudes toward reform and teacher preparation for implementing reform. 

Or ganization and Management The plan is to work through Education International, a union 
representing 20 million teachers and allied school personnel worldwide. A member of the association 
in each country will be asked to manage the country survey of teachers. Dennis Holmes and Marsha 
Levine are part of the team conducting the study. A person with recognized experience in conducting 
international studies will be invited to serve as a consultant to the project. 

The Institute will work with Education International to identify teacher organizations in 8-12 OECD 
countries to work with the Institute in conducting the study. A representative from the teacher 
organization(s) in each country, along with a contact person from the Minister or Secretary of 
Education's office will be asked to help the Institute implement the project. 

Participants 

Pilot test: Washington, D.C.-Virginia-Maryland metropolitan area teachers, staff 
development personnel, and the respective superintendents. 

International study: teachers and allied school personnel in 8-12 OECD countries. Schools 
involved will be jointly selected by the Secretary /Minister of Education and the Education 
International member organized in each country. 

SamnUi 

A contact person from the Minister or Secretary of Education's office will be asked to provide 
a list of teachers to be considered for possible inclusion in the sample population. The 
sampling plan will have to be designed to take into account the different ways that school 
systems are organized in each of the participating countries. The Institute will work with the 
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contacts in each country to design an appropriate sampling plan and the actual survey 
instrument. 



Procedures and Summary of Content A study on teachers* professional development and its 
relationship to implementing education reforms is presently being piloted. This study is being 
conducted in the United States and involves the eight school districts in the Washington, D.C.- 
Virginia-Maryland metropolitan area. In particular, the study will attempt to determine wlkether 
education reform is occurring in these districts and whether teachers' professional development 
programs have been (re)designed to support the reform efforts. It will involve teachers, staff 
development personnel, and the respective superintendents. 

In March 1994 Mary Futrell chaired a meeting convened by the Education International of leaders of 
all of its member organizations in OECD countries to discuss education reform. In June, Education 
International reconvened the same group and invited the Ministers/Secretaries of Education from the 
OECD countries to meet with us to explore major issues related to education reform in more detail. 
The contacts made and information gatiiered as a result of the meeting with the teacher leaders and 
the Ministers/Secretaries will be used as part of the international study. The pilot study results will 
be used to fmalize design of the prototype for the George Washington University international study. 
A full proposal for the study will be developed during the sununer of 1994. 

Timetable 

1993 The exploratory phase was completed; a proposal for the study was developed. 

1994 A study in the United States on teachers' professional development and its relationship 
to implementing education reforms is being piloted. 

1994 A full proposal for an international study is being developed. 

Funding Federal agencies and/or foundations will provide funding. 

Information Sources 

Mary Hatwood Futrell, Director 

Institute for Curriculum, Standards and Technology 

George Washington University School of Education and Human Development 

2201 G Street, N.W. 

Washington, D.C. 20052 

telephone: 202/994-2304 

facsimile: 202/994-7207 



FutrelU Mary Hatwood 

1993 Letter to Dorothy Gilford, director. Board on International Comparative Studies in Education. 
October 28. 

1994 International comparative study of teacher attitudes toward reform and teacher preparation for 
inq[)lementing reform. Status report for Board on International Comparative Studies in 
Education. April 20. 
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NOTE: A draft of this study summary was reviewed and edited by Mary Futrell at George 
Washington University in Washington, D.C. on July 11, 1994. 
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LANGUAGE EDUCATION STUDY 



International Organization International Association for the Evaluation of Educational 

Achievement (lEA) 

Years of Data Collection 1995: National policy profiles 

1997: Case studies and survey of students, teachers, and schools 

Purpose Education in languages is important everywhere in the world, and its importance increases 
as societies change, assess their expectations for what schools can do, and evaluate their economic 
positions internationally and human resources nationally. Major transformations in how languages are 
taught; used in work and daily life; and valued for all aspects of personal, academic, business, and 
societal opportunity have occurred within the past 20 years. 

Education in foreign and second language is important throughout the world. Some global trends that 
make languages in education especially prominent at this time are: 

0 Restructuring of international economic and political agreements require conunon, facilitating 
languages as well as new priorities in the languages needed by specific nations to realize these 
relations through workers' skills, for labor mobility, cooperation, and awareness across 
cultures. 

0 Most academic, technical, scientific, and financial information is now being coded into a few 
languages, making knowledge of these languages necessary to realize social, intellectual and 
economic advances in all parts of the world. 

0 Increased global migration has created massive demographic shifts between so-called 

developed countries and from rural to urban areas, making language education for migrant or 
inunigrant populations increasingly essential for social participation and harmony as well as 
occupational opportunities. 

0 The foregoing trends are prompting many societies to lose the language resources of 

indigenous or minority populations, resulting in increased calls for public institudons to 
accommodate and capitalize on ancestral heritages and multi*linguistic diversity. 

Comparative information and research to assess these trends are needed by: 

0 policy makers who have little current data at their disposal to determine the effectiveness or 

impacts of existing programs 
0 business and industry that need reliable indices to determine or forecast current capacities for 

future needs 

0 educators who lack a systematic data base on language education to raise awareness of 

curriculum or instructional choices 
0 consumers of language education who lack choices about language options and instructional 

programs in the absence of clear*cut information 
0 educational researchers who lack empirical data to form a comprehensive international 

perspective of educationally-relevant theories and recommendations for teaching practices of 

school policies in language education 

The lEA Language Education Study will address these issues systematically in order to develop a 
comprehensive international information base focused on four purposes: 



IntematioMl Com{MUitiv« Studies - 65 



71 



Winter 1994-95 



0 To determine the yield of language curricula in different countries. What percentage of 

students reach a basic, threshold level of proficiency in specific languages? What percentages 
reach an advanced, fluent proficiency? What differences in yield exist between and within 
countries? 

o To describe the scope and content of language curricula in different countries. What are the 
key features of foreign and second language curricula (number of languages taught, levels of 
student participation, starting ages, duration, intensity)? What exemplary possibilities for 
foreign and second language education exist? 

0 To identify the factors related to differences in yield in language curricula. To what extent 
are the difference in yield at the national level, the school level, and the individual level 
related to differences in curricula for language education? What is the impact of out*of*school 
opportunities on the learning and uses of foreign and second languages? To what extent do 
such relations between yield, curriculum factors, and environmental factors vary within and 
between countries? 

0 To assess needs and promising options for changes in foreign and second language curricula. 
Is the yield of language curricula sufficient for societal purposes in specific countries and 
educational jurisdictions? Which aspects of foreign and second language curricula call for 
change? 

There is no precise, authoritative information on what language education is accomplishing around the 
world. Few countries know accurately what their own systems for language education are doing, how 
successful their language programs are in terms of national priorities, how useful they are for which 
types of students or purposes, or how they may compare to those of neighboring countries. The IE A 
Language Education Study is an effort to gather this information, which will be useful for educational 
policy and practice planning^ and for socio-economic planning as well. 

This comparative survey of language education will: 

0 Determine what school systems around the world do and produce in language teaching and 
learning. 

0 Develop international assessment standards and tests to define basic and advanced levels of 

communicative competence in key languages. 
0 Describe the scope and content of school language curricula as well as societal conditions 

directly related to language learning. 
0 Identify optimal configurations for school language learning appropriate to national priorities 

and situations. 

0 Describe exemplary cases of school language teaching and learning, along with noteworthy 

innovations internationally. 
0 Assess needs and recommend promising options for change in participating countries. 

This study will provide: 

0 overview information on national policies for education in foreign, second, and minority 

languages in about 60 countries 
0 descriptions of language curricula based on surveys of students, teachers, and schools 

representative of secondary education in about 30 countries 
0 descriptions, using internationally-validated tests, of the proficiency for conununication that 

students in about 30 countries achieve in English, French, German, and other commonly 
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taught languages at (1) the end of the compulsory schooling and (2) upon completing 
secondary education 

0 case studies of exemplary practices in language teaching and learning in about 15 countries 
0 recommendations of needs and promising options for change in school language curricula for 
each participating country 

The study will help participating countries to promote the value of functional communicative 
competence in key languages^ capitalize on resources and the highest levels of research expertise 
coordinated internationally, as well as learn about and critically analyze their own language education 
systems and current conditions for language learning. Results will be useful for governments, 
^ucators, businesses and industries, and researchers, as well as language students and their families. 

Organization and Management The National Foundation for Education Research in England and 
Wales has been designated as the international coordinating center. This designation was made 
through a process of sending the proposal out for bids by potential international coordinating centers. 
The international coordinating center has the responsibility to coordinate the study and for the fmal 
results and reports, which includes: research design (including the time line and analysis plan), 
instrument development, data analysis, and r^rting. Additional responsibilities that are generally 
conrnion to all lEA studies (fund raiaing, field operations [including sampling procedures, 
standardized manuals, data collection, ^raining of national research coordinators' field monitoring], 
and data management) are the responsibility of the international coordinating center or the lEA 
headquarters, as determined by mutual agreement and approved by the Language in Education Study's 
International Steering Committee and the lEA Standing Committee. 

The lEA Standing Committee will define how the international coordinating center will coordinate 
with lEA headquarters; negotiations between the ICC and headquarters will determine how and by 
whom data will be collected and who will have responsibility for other aspects of the study. Either 
the ICC or the lEA headquarters may contract out activities. Other responsibilities are to be in 
accordance with lEA by-laws (lEA Guidebook 1993, Section 8, Appendices pages 173-175). 

The international coordinator is located at and manages the international coordinating center. He is 
responsible for the international aspects of the study, is the senior research officer of the study, is 
responsible to the General Assembly and Standing Committee, serves ex officio on the International 
Steering Committee, and oversees the work of national research centers. The coordinator has 
responsibility to maintain conununication about the study with all participating countries, the 
International Steering Committee, and other interested persons; provide documentation on all basic 
components of the study; manage the international funds of the study according to budgets and other 
policy decisions of the General Assembly; arrange for and act as rapporteur of meetings of the 
International Steering Committee and the International Project Committee; arrange meetings of 
national project coordinators and other meetings associated with the study; ensure agreed deadlines for 
the study are met by all participants; arrange for conduct of all phases of the study; arrange for 
incorporation of basic cross-system data into suitable archives for secondary analysis; and arrange for 
planning and writing of all international project reports and publications. 

The international coordinator has responsibility to appoint a project manager to manage all stages of 
the study and appoint a data manager to organize the international data processing centers. The 
international coordinating center has responsibility to: prq)are all instruments, manuals, codebooks, 
and documentation and check all constructs at the pilot suge; monitor sampling in cooperation with 
the sampling referee; arrange for the continuous training of national project coordinators; prepare data 
entry, cleaning and merging programs to be used by the national research centers; prepare and test all 
programs required for international statistical analyses; check incoming data sets and documentation; 
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document all stages of data processing procedures; ensure printouts can be easily read and interpreted; 
prepare graphics and software and secure appropriate hardware services; and provide each 
participating center with appropriate documentation and software. 

A Steering Committee has been established for this study. Its membership includes Alister Gumming 
(chair), OISE, Canada; Francis Debyser, CIEP, France; Kees de Glopper, University of Amsterdam, 
The Netherlands; Elana Shohamy, Tel Aviv University, Israel; John Trim, Council of Europe; G. and 
Richard Tucker, Carnegie Mellon University, U.S.A. 

Pfsign 

Participants Twenty-five lEA member countries have expressed interest in participating in 
the study. Some are interested in participating in regard to foreign languages widely used for 
international conmiunication. Fur^er interest exists for optional studies of education in 
second languages (education in a majority language for ethnic, inmiigrant, or regional 
minorities on a specific country). This interest focuses on education for social interaction and 
academic studies and will address diverse populations as well as complex factors in local 
societal contexts. 

The United States (as a whole nation or selected states) might choose to participate in the 
whole study or some of four parts of the study (national language education profiles; survey 
of student language proficiency for two populations; survey of language teaching and learning; 
case studies of exemplary, successful schools). This decision would depend on the nature and 
focus of the study (for example, which language/s is/are included) and on available funding. 
Most countries want to focus on English; the United States would be interested in Spanish as 
a foreign language, and in English as a second language. 

Sample Simple, but sophisticated sampling techniques will be used, to keep numbers of 
students and schools involved to a minimum while achieving representative sample 
populations. At the first stage of sampling, schools will be sampled with probability 
proportional to their size, at both population levels. At the second stage of sampling students 
will be sampled randomly within schools. There will be no sampling of intact classrooms; 
consequently school effects and classroom effects cannot be separated. The study will aim for 
an effective sample of 400 students for those proficiency measures that will be collected from 
all students (i.e., reading and listening comprehension). The precision of estimates of 
population values will thus be + or - 5 percent of a standard deviation for any given variable. 

For other skill domains (typically writing and speaking) students will be sub-sampled 
(typically 5 students per school per skill); the effective N aimed for will be 200 students. 
Between 20 and 30 students will be tested per school. Stratification will be employed in the 
preparation of the sample designs, in order to enhance sampling accuracy. Students will be 
tested in one language only. Countries testing multiple languages may opt for separate 
samples of schools, one school sample for each language, or for sampling separate student 
groups within a conunon sample of schools. 

Procedures and Summan of Content The study will assess language proficiency as the capacity to 
use a particular language appropriately and accurately to achieve social purposes in particular contexts 
(both inside and outside of school). It will address various competencies and domains of language 
use, rather than limiting itself to traditional testing of students* knowledge of grammar and 
vocabulary. Assessment will consist of focused and integrated tasks (tasks that require students to 
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communicate interactively, such as reading and writing in conjunction, or speaking and listening in 
ongoing conversations). 

School policies for foreign language education can be readily identified and compared internationally, 
and they form the principal focus for the of this study. School policies for second language education 
are considerably more complex, interact with local social contexts, and vary in curriculum 
realizations, which makes tiiem suitable as optional components of the study according to interests and 
situations of specific countries. 

Previous research points the way to many learning and curriculum variables, design issues, test 
characteristics, and sampling requirements that this study can adopt. For example, empirical studies 
of students' language performance have generally found variation according to such factors as 
situations of language use or communication tasks; previous education or literacy; age factors; time 
spent on language learning; structures or features of particular first and second languages; attitudes 
toward the society or culture of the language being learned; and the status and use of the languages in 
the local community. 

Numerous curriculum and instructional variables have also been analyzed, though few have received 
rigorous, large-scale empirical evaluation: instructional approaches or didactic procedures; 
curriculum content and organization; and effects of specific media, technologies, or resources for 
language learning. 



The core study and options design is as follows: 

Core Study Options 

Foreign Langxiage Education Second Language Education 



Population: 

15- md 18-year-old students 
Communicative competence 



Curriculum yield 



Curriculum scope and content 



Population: 

9-year-oId students and adults 
Academic skills 



Survey of teachers, schools, 
students, and national situation 



Case studies of exemplary 
programs 



Effective factors analyses of 
relations between yield, 
curriculum, and content 

Options for change 



Survey findings 



Case study findings 



National profiles inventory: This first phase of the study will provide at-a-glance overviews of the 
general context and policies for language education in each country. Findings will identify key policy 
issues and factors in school contexts broadly influencing curricula and students* learning in specific 
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languages. A survey vespott form will be conceived, designed, pilot-tested, and then distributed to all 
participating nations, as well as other nations that (because of limited resources or interest) will 
engage in this part of the study only. Data collection (scheduled for January 1995) will be reported 
initially (scheduled for November 1995) in the form of an inventory booklet; then a more extensive 
compendium reference document will be published the following year. 

Language proficiency assessment: A major portion of work during the study will be devoted to 
developing, conducting, and analyzing results from assessments of student language proficiency - all 
coordinated centrally through the international coordinating center. The study will produce and 
validate a common set of tests to assess the proficiency that students achieve to communicate in 
speaking, listening, reading, and writing in key languages, as selected by each country. Results will 
indicate the yield of school systems overall in terms of the percentage of students achieving either (1) 
threshold or (2) fully proficient levels of communicative competence at (1) the end point of 
compulsory schooling and (2) the completion of secondary schooling. 

Descriptions of language curricula: Work on questionnaires for school administrators, teachers, and 
students will follow a schedule similar to that for language proficiency testing. C^estionnaires will 
survey representative samples of students, teachers, and schools to describe their characteristics and a 
wide range of resources and practices related to language curricula and learning - assessing the fit 
between resources available and those most frequently used in practice in each country. After 
developing and pilot-testing instruments in 1996 and data collection in 1997, results will be reported 
in booklet form (scheduled for early 1998), followed by national reports and a full international report 
(later that year), incorporating results of language assessments as well. 

Case studies: To assure that this aspect of the study is sensitive to the interests of participating 
countries, initial consultations will review options for case study inquiry (classroom observations, 
ethnographic interviews, narrative techniques, teacher action research), determine common analytic 
purposes and frameworks, and link the design to issues highlighted in the national profiles inventory 
for each participating country. Case studies will provide vivid, holistic portraits of exemplary 
practices in language teaching and learning. Illuminating in depth what actually happens in 
innovative, successful classrooms, this aspect of the study will provide educators widi concrete, 
practice^based models of the experiences and conditions of language education. Building on initial 
results from surveys and proficiency assessments, case studies will be conducted in certain countries 
(scheduled for 1997), which will produce a booklet, then a detailed compendium of results (1988) 

Language proficiency yield, optimal configurations of language curricula, needs and options for 
change: In the final year of tiie study, data analyzed from the national profiles inventory, language 
proficiency assessments, descriptions of language curricula, and case studies will be compiled and 
evaluated centrally to produce tables and reports on language proficiency yield, curriculum 
descriptions, and optimal configurations of language curricula. Project centers will prepare national 
reports as well as reports on needs and promising options, ensuring they are relevant to local 
situations. Analyses, preparation of major books, writing of articles, and public presentations will 
proceed intensively through this final year but continue, according to the interests of participating 
researchers, for several years following the completion of the project. 

Information from the surveys, testing, and case studies will be synthesized to identify - in respect to 
the goals and societal contexts of each participating country - optimal configurations for school 
language education, specific needs in each country, and promising options for change. National 
reports are scheduled for publication in late 1988. An overview booklet will identify (1) key 
variables amenable to manipulation within educational systems to improve language achievement as 
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well as (2) major differences in societal contexts between countries that affect language acquisition 
and use. 

Timetahle 

1993 An amended study proposal was approved by the lEA General Assembly. 

A specific set of responsibilities for the international coordinating center and 
procedure for agreement between the ICC and lEA headquarters on other 
responsibilities related to the study approved by lEA General Assembly was 
determined* 

1994 The intemationai coordinating center and international coordinator were designated 
through competitive bidding. 

Funds are being raised by lEA headquarters. 

A Steering Conmiittee was established. 

Member country participation will be established. 

National Project Coordinators will meet O^te 1994). 

The study design will be discussed by the lEA Standing Committee. 

Instruments and procedures will be developed. 

The proposed timetable is likely to be altered by a decision of the General Assembly.* 
(August) 

1995 National policy profiles data collection is scheduled. (January) 

Initial reporting of policy profiles data collection is scheduled. (November) 

1996 Survey of students, teachers, and schools pilot testing instrument will be developed. 

1997 Survey of student, teacher, and school data will be collected. 

Case studies will be conducted in certain countries. A booklet of case studies will be 
published. 

1998 Results of the survey of students, teachers, and school will be reported in a booklet, 
(early 1998) 

National reports and a fiill international report on the survey of students, teachers, and 

schools, including language assessments, will be published. G^ite 1998) 

A detailed report of results of case studies will be published. 

National rq>orts of a synthesis of information from the survey, testing, and case 

studies will be published, (late 1998). 

It is expected that the first year of the study will be confined to work on the national 
profiles inventory, with the survey element postponed for one year. 

Publications Case studies will be reported first in a booklet, and then in a detailed report of the 
results. The survey of students, teachers, and schools will be reported first in a booklet, and then in 
national reports and a full intemationai report, which will include language assessments. National 
reports of a synthesis of information from the survey, testing, and case studies will be published. 

Funding lEA headquarters is seeking funding from foundations, intemationai organizations, and 
countries other than the United States. The United States will not be a principal or major fimder of 
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the coordinating center or international activity. The National Center for Education Statistics may 
provide a small amount for the U.S. national center* 



Information Sources 

Peter Dickson» International Coordinator 
National Foundation for Educational Research 

in England and Wales 
The Mere, Upton Park 
Slough, Berkshire SLl 2DQ ENGLAND 

telq>hone: 44-7-53-74-12-3 

facsimile: 44-7-53-69-16-32 

Internet: EZUE100@VMSFE,ULCC.AC.UK 

International Association for the Evaluation of Educational Achievement Secretariat 
14 Sweelinckplein 

NL2519 The Hague, THE NETHERLANDS 
telephone: 31-70-346-96-79 
facsimile: 31-70-360-99-51 
e-mail: IEA@SVO,NL 

Gordon M . Ambach, U.S. Liaison to lEA 
Council of Chief State School Officeis 
One Massachusetts Ave., N.W., Suite 700 
Washington, D.C. 20001-1431 

telephone: 202/ 408-5505 

facsimile: 202/ 408-8076 

e-mail: gambach@nas.edu 



International Association for the Evaluation of Educational Achievement 

1993 Activities, Institutions, and People: lEA Guidebook 1993-94. lEA, The Hague, The 
Netherlands. 

1993 The lEA Language Education Study Proposal. July. For discussion, revision, and approval. 
34th General Assembly, El Escorial, Spain, September 1993. lEA, The Hague, The 
Netherlands. 

Ambach, Gordon M. 

1993 lEA Language Education Study. Status reports presented to the Board on International 
Comparative Studies in Education. January 28, June 19, November 7. Council of Chief State 
School Officers, Washington, D.C. 

1994 lEA Language Education Study. Status reports presented to the Board on International 
Comparative Studies in Education. February 6, May 16. Council of Chief State School 
Officers, Washington, D.C. 

Selden, Ramsay 

1993 Activities on Foreign/Second Language Study. Status report prepared for the Board on 
International Comparative Studies in Education, June 2. Council of Chief State School 
Officers, Washington, D.C. 
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NOTE: This study summary was reviewed and edited by Peter Dickson at the National Foundation 
for Educational Research in England and Wales (NFER), in Berkshire, England on July 11, 1994. 
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iV£ W STANDARDS PROJECT (benchmark activities) 



International Organization Learning Research and Development Center at the University of 

Pittsburgh and the National Center on Education and the Economy 

Years o f Data Collection Ongoing from 1993 

Purpose The New Standards Project is a voluntary association committed to the joint development of 
new standards and new examination systems for U.S. schools. The goal is to improve student 
performance radically by using standards and assessments as instruments for comprdiensive and 
systemic reform of American education. The 25 state and district Partners in the New Standards 
Project are developers and users of a system of assessments based on portfolios, projects, and 
performance tasks, employing the content and performance standards defined by groups such as the 
National Council of Teachers of Mathematics. The Partners seek mutual association in order to link 
their own assessments to a shared national standard. It is intended that the standards used in the 
development of assessments will be internationally competitive. 

A process of collaborative benchmarking of education standards has been defmed by which several 
countries can compare their educational standards and expectations. This process is designed 
primarily to serve the New Standards Governing Board in its standards*setting process. The Board 
needs information from high achieving nations about their content and performance standards in 
mathematics, language arts, and science for key transition points in a child^s schooling career. With 
this information, the Board will be prepared to assure the New Standards Partners that they are 
linking to a system of internationally competitive content and performance standards. 

Organization and Management The New Standards Project co-directors are located at the Learning 
Research and Development Center at the University of Pittsburgh and the National Center on 
Education and the Economy. A coordinator for international benchmarking is located at the Learning 
Research and Development Center. The project is also served by a consultant. 

Design New Standards International Benchmarking is a case study. Data collection includes 
interviews with teachers and other education specialists, examination and translation of documents 
related to education, and collection of graded student work samples. 

Participants 

Australia, Great Britain, France, Germany, Japan, the Netherlands, Sweden, and the United 
States. New Standards is benchmarking U.S. madiematics standards/tasks/portfolios with 
mathematics education in Australia, France, Japan, the Netherlands, and Sweden; it hopes to 
add Germany and Great Britain. Other countries will be used in language arts and science. 

• 

Samtik 

Case studies. 



Procedures and Summary of Content The benchmarking model begins with data collection aimed at 
describing the standards and procedures used by appropriate counterparts in a field of endeavor. 
Rather than trying to convert all standards to a common metric, benchmarking describes what others 
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are doing in their own terms and within their own system of values. This is different from other 
comparative efforts that focus on how students perform on a test of conmion items. 

A carefully chosen set of questions about issues serves as the framework for collaborative 
international benchmarking. The New Standards process for collecting benchmark information 
engages a set of collaborators from several countries in generating information that can be used by 
educators and policy makers in each country. The common descriptors do not force an artificial 
structure on the systems investigated. Raiher, the common set of questions yields a description of 
each country that sets various elements of its goals and standards practice in a systemic context. 

The collaborative benchmarking model employs a set of common questions in order to obtain 
descriptive and contextualized comparisons of goals and standards. The result is a set of rich 
descriptions of the educational standards of each participating country. Once the standards are 
understood through the collaborative benchmarking research process, presentation formats can be 
customized to present decision makers in each country with the information they need to inform their 
own decision making. Decision makers include teachers and other education professionals, parents, 
legislators, other policy makers, and the general public. Benchmarking does not imply that goals and 
standards found desirable in other countries will necessarily be adopted; it holds only that the 
experience of other countries can be useful in setting goals and standards at home. 

Data Collection and Analyses The six organizing questions for the New Standards benchmarking 
process are: 

o What is the structure of the education systems in comparison countries? What are the key 

transition points and decision points for students? 
o What are students in other countries expected to know and be able to do at key transition 

points in their schooling careers? 
o What kinds of performances are used to demonstrate competence? 
o What counts as ''good enough** in these performances? 
o What portion of the cohort is meeting the standard? 
o What reform efforts are underway or on the horizon? 

These questions express the concerns that have motivated New Standards Partners to come together in 
an effort to link their state- and district-level assessments to a shared national standard: What is and 
is not working in other states? How much of what kind of work should we be requiring of our young 
people? What must the students in our state know and be able to do in order to take their places as 
citizens and workers? What does it mean to be educated at the turn of this second millennium? 



To determine the content standards other nations use in mathematics, language arts, science, and other 
subject areas, the benchmarking process examines standards clearly expressed in national or regional 
curricula as well as standards that must be inferred from texts, syllabi, and class work. Materials 
collected and analyzed for content standards include national, regional, or school curricula; legislative 
or ministerial directives related to content; commonly used textbooks; exams; and examples of 
classwork. Teachers and other education professionals in each partner country review the research 
analysis at each step of data collection and interpretation to provide assistance in understanding 
official documents and to fill the gap between those documents and actual classroom practice. 

Internationally benchmarking the New Standards assessment system includes comparing it with other 
assessment systems, where they exist, and otherwise with testing or examining practices. Materials 
collected and analyzed to understand the kinds of performances students around the world engage in 
to demonstrate competence are examples of external national exams (including information on when 
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they are given and for what purpose ) and examples of internal school or classroom exams and other 
evaluation activities. A professional review panel discusses and reviews these materials for 
understanding of the assessment system. 

To determine benchmarks for "best in class** performances in schools around the world, New 
Standards collects scored student work on exams and classwork, scoring rubrics or guides, and 
teacher commentary on what the scores mean. Although this information is difficult to collect, New 
Standards believes that collections of work samples will be the cornerstone of genuine internationally 
competitive standards. New Standards will encourage research partners to collect and compile student 
work when possible. In some countries, New Standards may have to do this work, although 
compilation from outside the country is less desirable. 

New Standards is committed to the ideal that all children in the United States can learn and that they 
can be held to higher standards of achievement. Benchmarks need to include information about what 
this ideal might mean in other countries; what portion of students are meeting the standards they have 
set? This information is gathered through data on pass/fail rates for national exams; where such data 
are not collected, the data result from professional estimates of percentages of students meeting the 
standards and percentages of students proceeding through various tracks, with clear descriptions of 
those tracks. 

To benchmark not only to current practice but to education reforms underway or being planned, the 
following information is needed from each of the countries chosen for benchmarking: plans currently 
mandated by the ministry of education, either in the development or implementation stage; "best 
guesses'* from university and research personnel; **best guesses'* from university and research 
personnel regarding the direction of change; and research and development work conducted by 
companies developing texts and other curricular materials. 

As American states are working with their districts to articulate both content and performance 
standards and to benchmark against internationally competitive mathematics achievement, the leading 
states are raising expectations for their own students. In an international environment as open to 
observation and communication as that of developed economies, we may expect that successful 
American efforts to improve student performance will draw as much attention from other nations as 
their efforts have hitherto drawn from us. This mutual interest sustains the collaborative mode of 
New Standards research and promises that benchmarking, undertaken to meet the needs of New 
Standards Partners in the United States, will remain international in scope. 

Timetable 

1994 The first mathematics reports will be published. (June) 

1995 Revised mathematics reports will be published. (June) 
The first language arts reports will be published. (June) 

1996 Revised language arts rq)orts will be published. (June) 
The first science reports will be published. (June) 

1997 Revised reports on science will be published. (June) 
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Publications 



R^rts on mathematics, language arts, and science will be published. 

Setting standards across borders, OECD Network A sununary volume. Lauren Resnick and 
Katherine Nolan. 

Possibilities for making New Standards Project benchmark activities materials available for 
public use are to be determined. 

Funding Funduig is provided by The Pew Charitable Trusts and the John P. and Catherine T. 
MacArthur Foundation, and by Partner contributions. 

In formation Sources 

Lauren B. Resnick, Co-director 
Learning Research Development Center 
University of Pittsburgh 
3939 O'Hara St., Rm. 824 
Pittsburgh, Pennsylvania 1S260 

telephone: 412/ 624-7485 

facsimile: 412/ 624-3051 

Marc Tucker, Co-director 

National Center on Education and the Economy 

39 State Street, Suite 500 

Rochester, New York 14612 

telephone: 716/ 546-7620 

facsimile: 716/ 546-3145 

Katherine Nolan, Coordinator for International Benchmarking 
Learning Research and Development Center 
3939 O'Hara Street 
Pittsburgh, Pennsylvania 15260 

telephone: 412/ 624-8319 

facsimile: 412/ 624-1470 



Resnick, Lauren B., and Katherine J. Nolan 

1994 Setting standards across borders. Organisation for Economic Co-operation and Development, 
Network A summary volume, (submitted) Organisation for Economic Co-operation and 
Development, P&ris, France. 

Resnick, Lauren B., Katherine J. Nolan, and Daniel P. Resnick 

1994 Benchmarking Education Standards. Prepared for the Conference on Uses of Interaattonai 
Education Data, convened by the National Research Council Board on International 
Con^arative Studies in Education. February. 
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NOTE: This study summary was reviewed and edited by the office of Lauren Resnick at the 
Learning Research and Development Center in Pittsburgh in May 1994; Katherine Nolan provided 
database information on December 6, 1994. 
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PREPRIMARY PROJECT 



International Organization International Association for the Evaluation of Educational 

Achievement 

Years of Data Collection Phase 1: 1989-91 

Phase 2: 1991-93 
Phase 3: 1994-96 

Purpose In countries throughout the worlds demographic, social, and economic changes are driving 
an accelerating demand for early childliood services. At the same time, a growing body of research 
is demonstrating the long-term benefits of high-quality early childhood programs, thus increasing the 
general awareness of the importance of children's early years. These developments point to a need 
for sound and adequate information on which to base early childhood services. The lEA Preprimary 
Project is a response to the dearth of information about children's care and education experiences 
prior to formal schooling. 

Some key questions inspired by current trends in early childhood care and education that have dictated 
the objectives and design of each phase of the Preprimary Project are as follows: 

Phase 1: 

o What kinds of early childhood care and education services do families use in various 

countries around the world? 
o What are factors (socioeconomic, cultural) that influence parents' choice of early 

childhood care arrangements for their 4-year-old children? 
o What are the characteristics of the early childhood care and education settings parents 

choose for their 4-year-olds? 

Phase 2: 

o What is the 'Equality of Life** for 4-year-old children in such settings as pre-schools, 

child care centers, and family day care homes? 
o How do adults' educational values and expectations for 4-year-olds affect the way they 

(parents, teachers^ caregivers) organize environments and activities for these children 

and the way they interact with these children? 
0 In any given nation, how does the "quality of life" for 4-year-old children in early 

childhood settings compare with that of 4-year-old children in early childhood settings 

in other nations? 

0 What is the relationship between children's developmental status and specific 
structural features and/or interactional processes of care/education settings? 

Phase 3: 

o What is the relationship between 4-year-old children's experiences in care/education 

settings and their later developmental status? 
o What role might early childhood care and education play in preparing children for 

formal schooling? 
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The approach of the project has also been governed by methodological considerations about the study 
of young children and by the multi-disciplinary perspectives that guide public policy in the area of 
early childhood services. An additional consideration has been the relative lack of knowledge about 
preprimary service as comparea with knowledge about services at other educational levels (elementary 
and secondary school). 

To guide public policy formulation, each participating country's national study will provide timely 
information tailored to address that country's most salient issues and gaps in knowledge. In addition, 
the cross^national analyses will provide a valuable comparative baseline. By examining a greater 
range of variation in early childhood experiences than occurs within any one country, researchers will 
be able to test the cross*cultural generalizability of country*specific explanations of child development 
and early school achievement* The Preprimary Project is designed to extend scientific knowledge 
about early socialization and to provide data to evaluate early childhood models aimed at promoting 
the wellbeing of children* 

Organization and Management Fifteen lEA member countries have participated in one or more 
phases of the Preprimary Project. Each participating country has established a national research 
center, s^pointed a national research coordinator to supervise the study, and assembled additional staff 
to coordinate data collection and analysis. In its capacity as international coordinating center, the 
High/Scope Foundation has appointed a research team to provide general supervision of project 
activities and to provide technical assistance to participating countries. Dr. Leslie Kish, a noted 
expert in the field of international sampling, works cooperatively with the participating countries and 
the ICC staff to provide technical assistance with sampling issues. 

The international coordinator, dq)uty international coordinator, and international project center are 
located at High/Scope. The international coordinator chairs the international project commiuee, 
which is comprised of those who put the study into operation (national research coordinators, 
international steering conmiittee, and international coordinator). National research coordinators meet 
once a year, or sometimes more frequenUy* 

Design The three interrelated phases of the Preprimary Project are designed to: 

0 Identify and describe the types of early childhood care and education settings used by 

the families with 4-year-old children 
0 Describe the processes that occur within the context of various types of early 

childhood settings 

0 Look at the differential effects of these settings on children's later development. 
Participants 

Phase 1: Belgium (French), China (PRC), Germany (Federal Republic), Finland, Hong 
Kong, Italy, Nigeria, Portugal, Spain, Thailand, and the United States* 

Phase 2: Belgium (French), China (PRC), Finland, Greece, Hong Kong, Indonesia, Ireland, 
Ita«y, Nigeria, Poland, Romania, Slovenia, Spain (Catalonia), Thailand, and the United States. 

Phase 3: Same countries as in Phase 2* Follow-up of children at age 7, who were observed 
in Phase 2 at age 4. 
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Phase 1: 

In each participating nation, researchers identified a probability sample of children 3 1/2 to 
4 1/2 years old and interviewed their families using a household survey questionnaire. In 
most countries, multi-stage stratified cluster sampling procedures were used. The Riase 1 
achieved sample sizes ranged from 424 in Belgium (Fr.) to 12,835 in China (PRC). In 7 of 
the 11 participating countries (China, Finland, Hong Kong, Italy, Nigeria, Thsuland, United 
States), response rates were 80% or higher. 

Phase 2: 

The population of settings included the major types of settings used by families (as indicated 
by Phase 1 data or by other information sources) and settings of particular interest to policy 
makers. It was expected that each participating country would identify at least two major 
types of extra-familial settings for the study. In addition, whenever feasible, countries would 
include familial care as a type of setting (to include children receiving only this form of care). 

To achieve levels of statistical power in data analyses commensurate with likely magnitudes of 
effects, researchers needed to include a sufficient number of settings. Preliminary 
considerations suggested that it would be sufficient to sample a minimum of 24 settings of 
each major extra-familial type, assuming that the total number of children attending each type 
of extra-familial settings sampled was at least 96. Within each setting selected, researchers 
randomly chose one or more children for inclusion in the study. 

Procedures and Summary of Content 

Phase 1: This phase was conducted between 1987 and 1991 and consisted of a household survey of 
families with 4-year-old children. Researchers in each of the 1 1 Phase 2 countries interviewed a 
representative sample of families. In doing so, they used the joint-national project directors- 
developed instrument, the Parent/Guardian Interview, which asked about the types of formal and 
informal care arrangements families use for their 4-year*olds. 

Other information collected in the interview included the n"mber of different settings attended by 
children, the reasons families use care/education services, the amount of time children spent in tfieir 
various settings, family satisfaction with services, family background, and a detailed description of a 
typical day in the child's life. The design of Phase 1 allowed researchers to establish a typology of 
settings and to examine the patterns of use of various settings and to examine the patterns of use of 
various settings as they relate to a number of variables, such as urban/rural residence and family 
characteristics. 

Funding to complete the Phase 1 data analysis and to prq)are the fmal report were delayed and the 
timdine was revised by the steering committee and international coordinating center. The fmal report 
will be published in Fall 1994 jointly by Pergamon Press and the High/Scope Press. 

Phase 2: This phase of the Pre{>rimary Project is called the Quality of Life Study. It is an 
observation study that nationally and cross-nationally examines the characteristics of settings 
experienced by 4-year-old children and the relationship between the children's developmental status 
and their experiences in these settings. The IS countries participating in Phase 2 observed the major 
settings identified in their national Phase 1 Preprtmary Project. 
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Three key types of setting processes were addressed through on-site observations during Phase 2: 
adults' organization of the children's time, children's activities, and adults' bdhaviors. The children 
observed in each setting were assessed with respect to developmental criteria O^guage development, 
fme-motor coordination). 

Along with the observation and assessment data, interviews and questionnaires were used to collect 
several other types of information, including: structural characteristics of settings (such as adultH:hild 
ratio, group size, management features, staff qualifications, and materials and equipment); family 
background data (such as socioeconomic information and hous^old composition); and adult 
perceptions (such as educational values and expectations for children). 

Phase 3: The third and final phase of the Prq>rimary Project is a follow-up study of the children 
observed at age 4. The 4-year-old children selected for observation and assessment in Phase 2 will be 
reassessed at age 7, the age when children from all participating countries will have completed at least 
a year of primary school. The evaluation at age 7 will draw on many sources to gather information 
about the children, including school records, interviews with teachers and parents, and assessment 
tests. Through the use of a wide variety of data sources, it will be possible to determine the effects 
of children's experiences in various settings during the preschool years. 

Four general areas will be assessed in Phase 3: 

o child developmental status (including cognitive, language, social/emotional, and 

academic) 
o family background 
o teacher and classroom characteristics 
o national curriculum 

Using me^^ures and guidelines from national research coordinators, the international coordinating 
center has developed measures for each of the areas. Participating countries are pilot-testing these 
measun ^ between January and June 1994 with final measures to be determined in Fall 1994. 

Data Collection and Analyses 

Phase 1: Researchers in each participating country used housdiold survey procedures to gather data 
directly from families with preschcol-aged children about their use of early childhood care and 
education services. The data were coded and sent to the international coordinating center where the 
cross-national data analyses were completed. The finding volume containing a wealth of descriptive 
information about the use of early childhood services in various countries is being published in Fall 
1994. 

Additional rq)orting activities included presentations at the 1990 annual meeting of the American 
Educational Research Association meeting and at the 1992 annual meeting of the National Association 
for the Education of Young Children. 

Phase 2: Data collection consisted of the following components: 

o In each early childhood setting selected, observers used the 3 observation systems to 

collect data on two nonconsecutive mornings, 
o The teacher/caregiver in each selected setting completed a setting-information about 

their beliefs about the importance of specific areas of development for young children 

(expectations). 
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o Data collectors administered 2 to 5 Child Developmental Status measures to each child 
and administered a family background interview and a "beliefs'* questionnaire to the 
parent/guardian of each child. 

With the data obtained through observations, assessments, interviews, and questionnaires, 
researchers are evaluating the effects of different types of care settings on child development 
by exploring the relationships among the structural features of settings, the socialization 
processes occurring in settings, various features of the child's background, and the child's 
developmental status. 

An early Phase 2 reporting activity was conducted at the 1989 annual meeting of the National 
Association for the Education of Young Children in a symposium co-sponsored with Organisation 
Modiale pour Teducation Prescolaire (OMEP). The Phase 2 findings will be r^rted in a set of six 
monographs: 

#1 Teacher/parent beliefs about children's development 

#2 Provider survey findings presenting data bout the characteristics of group settings in 
various nations 

#3 and #4 

Observation system findings. Observation system findings will be the major results 
used to develop a new view of "quality" in early childltocd programs, a view based 
on process characteristics such as adult/child interactions and ad'^U involvement. The 
monographs will present findings about the amount of time children spend in different 
areas of learning, how adults interact with children, and who determines how children 
spend tiieir time in early childhood settings. Findings will be available for various 
types of U.S. settings and for different types of settings in other nations, producing a 
very rich body of data for early childhood professionals and policy makers. 

#S Child assessment findings from each participating nation will be reported. In the 
United States, these findings will provide information related to the first of the six 
U.S. national educational goals, which states that "by the end of the decade all 
children in America will start school ready to learn." Phase 2 will provide 
information about the developmental status of children in many areas (cognitive, 
language, social competence) as they enter the formal educational system. With the 
study's design, researchers will be able to examine the relationship betweeu children's 
developmental status and their preschool experiences, in the United States and in other 
countries. 

#6 Causal models of the complete set of findings will present the major cross-national 
findings for Phase 2. 

In connection with Phase 2, the High/Scope Foundation is preparing a set of 15 videot^es that 
illustrate the variety of early childhood settings encountered in various Phase 2 participating countries. 
Each of the 15 countries is represented by a 45-minute videotape that describes the care/education 
settings available to families and the nature of children's experiences in each setting. The set of tapes 
will show how facilities, materials, teaching styles, and program philosophies vary from setting to 
setting, and from country to country. They will reveal how political, socioeconomic, and cultural 
differences worldwide impact young children's development. 
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Phase 2 preliminary data were r^rted at the 1993 annual meeting of the National Association for the 
Education of Young Children. The international coordinator and international deputy coordinator 
presented preliminary findings about teacher and parent beliefs about important areas of development 
for 4*yearH)ld children. 



Timetable 

1980 Researchers outlined ideas for an lEA study. 



1982 The lEA General Assembly approved the planning group's draft proposal; an 

international project steering committee was iqppointed; an international coordinator 
and a coordinating center were selected. 

1984 The international steering conunittee and international coordinating center completed 
the final proposal for the three-phase project; the General Assembly approved the 
proposal. 



1987-1991: Phase 2 

1989 The Preprimary Project co-sponsored a symposium at the NAEYC annual meeting. 




1990 National research coordinators and the deputy international coordinator made 
presentations at the AERA annual meeting. 

The international coordinating center and international steering committee presented 
Phase 2 (Quality of Life Study) and Phase 3 (Follow-up Study) workplan to the lEA 
General Assembly. (June) 

1991 Phase 1 data collection was completed. 
Phase 2 data collection was initiated. 

1992 Phase 1 national data analyses were completed. 

National project coordinators presented major Phase 1 findings at the NAEYC annual 
meeting. 

1993 Phase 2 data collection was completed in 13 of IS countries. 
Phase 1 cross-national data analyses were completed. 
Phase 3 measures were developed for pilot-testing. 

Phase 2 reports timeline and contents were revised. 

The intematbnal coordinator presented the Phase 2 report with policy implications 
and the Phase 3 proposal to the lEA General Assembly; the lEA General Assembly 
aqpproved the Phase 3 proposal. (Sq)tember) 
Phase 2 findings were presented at die NAEYC annual meeting. 

1994 The Phase 1 findings volume will be published by Pergamon Press and by High/Scope 
Press. 

Phase 2 national and cross-national data analyses are being conducted. 

VtiZTii 2 Monograph #1 - teacher and parent beliefs about children*s development will 

be published. 

Phase 3 pilot-testing is being conducted. 
Phase 3 data collection will begin* 
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1995 Phase 2 cross-national data analyses will be conducted. 

Phase 2 Monogr^h #2 - group-setting characteristics will be published. 

1996 Phase 2 Monographs #3 and #4 - observation system findings will be published. 
Phase 2 Moaogr^h #S - child assessment fmdings will be published. 

Phase 3 data collection will be complied* 

1997 Phase 2 Monograph #6 * causal models of the complete set of findings will be 
published. 

Publications 

How Nations Serve Young Children: Profiles of Child Care and Education in 14 Countries. 
Ypsilanti, Michigan: High/Scope Press. (1989) 

Families Speak: Early Childhood Care and Education in 1 1 Countries. Report of Phase 1 of 
the lEA Preprimary Project. Oxford: Pergamon Press and Ypsilanti, Michigan: High/Scope 
Press. (1994) 

Phase 2 fmdings report will be published 1994*95. 

Phase 2 Monograph #1 - Teacher/parent beliefs about children's development. (1994) 
Phase 2 Monograph #2 - Provider survey fmdings. (1995) 
Phase 2 Monographs #3 and #4 • Observation system fmdings. (1996) 
Phase 2 Monograph #5 - Child assessment findings. (1996) 
Database 

Name lEA Preprimary Project - Phase 1 

Description Information about families' use of early childhood services collected using 
household survey procedures with nationally representative samples of families in 1 1 
countries* Daia include types of services used, reasons for using services, cost, problems, 
and satisfaction with services. A portion of each country's file contains a detailed description 
of a 244iour period in the 4*year-old child's life, including where the child was and who was 
supervising the child. Finally, the data include family background information. (Countries: 
Belgium, China [PRC], Finland, Germany, Hong Kong, Italy, Nigeria, Portugal, Spain, 
Thailand, United Sutes) 

To whom available To be determined. 

Restrictions To be determined. 

whi^ means available To be determined. 

Charge to user To be determined* 



ERIC 
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When available January 1995 (Phase 1 of the 3-phase lEA Pr^rimary Project is completed. 
As the lEA international coordinating center for this project, High/Scope has prepared the 
database for Phase 1, and will send a copy of the database to lEA headquarters, where it will 
become part of the overall set of lEA databases.) 

Forfiurther information about access to this database, contact: 

Patricia P. Olmsted 

High/Scope Educational Research Foundation 
1 600 North River Street 
Ypsilanti, Michigan 48198 
telephone: 313/ 485-2000 
facsimile: 313/ 485-0704 

electronic mail: PAT_OLMSTED@UM.CC.UMICH.EDU 

Funding Each participating country is responsible for locating the funds and other resources 
necessary to conduct the Preprimary Project nationally. Support to the international coordinating 
center has been provided by government agencies (U.S. Department of Health and Human Services, 
U.S. D^artment of Education), the National Academy of Sciences, and foundations (Carnegie 
Corporation, High/Scope Foundation, Packard Foundation, Pew Charitable Trust, Spencer 
Foundation). 

Information Sources 

Dr. David P. Weikart, International Coordinator and International Project Committee Chair 
High/Scope Educational Research Foundation 
600 North River Street 
Ypsilanti, Michigan 48198-2898 

telq)hone: 313/ 485-2000 

facsimile: 313/ 485-0704 

Dr. Patricia P. Olmsted, International Deputy Coordinator 
High/Scope Educational Research Foundation 
600 North River Street 
Ypsilanti, Michigan 48198-2898 

telephone: 313/ 485-2000 

facsimile: 313/ 485-0704 

e-mail: Pat_01msted@UM.CC.UMICH.EDU 

International Association for the Evaluation of Educational Achievement Secretariat 
do SVO 

Sweelinckplein 14 

2517 GK The Hague THE NETHERLANDS 
telephone: 31-70-346-96-79 
facsimile: 3 1-70-360-9? -51 
e-mail: iea@svo.nl 
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Gordon M. Ambach, U.S. Liaison to lEA 
Council of Chief State School Officers 
One Massachusetts Ave., N.W., Suite 700 
Washington, D.C. 2000M431 
telephone: 202/ 408-SSOS 
facsimile: 202/ 408*8076 
e-mail: gambach®nas.edu 



Bridgman, A. 

1989 The lEA Pieprimary Project: How a dream became reality. Childhood Education. 65:157- 
1S9. 

High/Scope Educational Research Foundation 

1988 lEA Prcprimary Project International Study: Overview, Parent/Guardian Interview, Sampling 
Information, and Funding Information. High/Scope, Ypsilanti, Michigan. 

1990 lEA Preprimary Project. Status reports prepared for the Board on International Comparative 
Studies in Education. David P. Weikart. September and December. 

1991 IE A Pr^rimary Project. SUtus report prepared for the Board on International Comparative 
Studies in Education. EUyn J. Zeve. April and September. 

1992 lEA Preprimary Project. SUtus report prepared for the Board on International Compr^rative 
Studies in Education. Patricia P. Olmsted. January, April, and September. 

1993 lEA Preprimary Project. Status report prepared for the Board on International Comparative 
Studies in Education. Shannon Lockhart. January. Patricia P. Olmsted. May and October. 

1994 lEA Preprimary Project. Status report prepared for the Board on International Comparative 
Studies in Education. Shanncm Lockhart. January and April. 

International Association for the Evaluation of Educational Achievement 

1990 lEA Preprimary Project Phase 2: QuaUty of Life Study and Phase 3: Follow-up Study 

Workplan. Submitted to the lEA General Assembly by the International Coordinating Center 
and International Steering Committee. June. 
1993 Activities, Institutions, and People: lEA Guidebook 1993-1994. lEA, The Hague, The 
Netherlands. 

International Association for the Evaluation of Educational Achievement and High/Scope Educational Research 
Foundation 

1986 Parent/Guardian Interview: U.S. Child Care Survey. lEA, Stockholm; High/Scope, 
Ypsilanti, Michigan. 

Olmsted, Patricia P. 

1992 A cross-national perspective on the demand for and supply of early childhood services. In 
Booth, A. (Editor), Child Care in the 1990s: Trends and Consequences. Hillsdale, New 
Jersey: Lawrence Erlbaum Associates. 

Olmsted, P.P., and K. Hoas 

1989 Preschool teacher training in Finland, Hong Kong, luly, and Thailand: Accounts from the 
lEA Preprimary Project. Childhood Education. 65:283-287. 

Olmsted, Patricia P., and David P. Weikart 

1988 lEA-Prepriroary Project. PresenUtton to the Board on International Comparative Studies in 
Education. November. 
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Olmsted* P.P.» and D.P. Wetkart» Editors 

1989 How Nations Ser/e Young Children: Profiles of Child Care and Education in 14 Countries. 

Ypsilanti» Michigan: High/Scope Press. 
1993 Families Speak: Early Childhood Care and Education in 11 Countries. Report of Phase 1 of 

the lEA Prcprimary Project. Oxford: Pergamon Press and Ypsilanti» Michigan: High/Scope 

Press. 



NOTE: This study summary was reviewed and edited by Patricia Olmsted at the High/Scope 
Educational Research Foundation in Ypsilanti, Michigan on June 24, 1994; Olmsted provided 
database information on December 1, 1994. 
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READING UTERACY STUDY 



International Orfanization International Association for the Evaluation of Educational 

Achievement 

Years of Data CoUection 1990 - 1991 

Purpose Because policy issues are of paramount importance for many of the educational leaders who 
support the lEA international studies, the Reading Literacy Study was designed to provide results that 
could serve as a basis for decisionmaking concerning the improvement of reading in primary and 
secondary schools. The study coincided with the UNESCO World Literacy Year and set 1990 as the 
year in which the first large-scale baseline data were established against which future achievements in 
literacy would be measured. This international study was designed to develop instruments for 
measuring literacy in each of Ui^ 32 participating country school systems. It was expected to establish 
a clear imified definition of literacy and to measure the comparative ability of educational systems to 
teach literacy skills. 

For this study, reading literal refers to the linguistic and cognitive processing of written language 
forms required by society and/or valued by the individual. As this implies, literacy is regarded as 
both a cultural practice and an intellectual achievement. For purposes of the study, literacy is defined 
as the ''ability to understand and use those written language forms required by society and/or valued 
by the individual.** This definition has been operationalized in terms of reading test items along three 
scales (narrative, expository, and document) that correspond to text types. 

The purposes of the lEA Reading Literacy Study are to: 

o Describe the achievement levds in reading literacy of comparative samples of students 

in education systems 
o Describe the voluntary reading activities of 9 and 14 year-old-students 
0 Identify differences in policies and instructional practices in reading, and study the 

ways in which they relate to students* achievement ix\d voluntary reading 
o Produce valid iriiiemational tests and questionnaires tha could be used to investigate 

reading literacy devdopment in other countries 
o Provide national baseline data suitable for monitoring changes in literacy levels and 

patterns over time. 

The products of the international study are: 

o valid measures suitable for measuring literacy across countries 

o estimates of the percentage of each population reaching specified levels of literacy 

o comparative data across countries of literacy achievement in three domains (nanative 

passages, expository passages, documents) on ?n international scale 
o identification of whidi school, teacher, and societal factors influence literacy and to 

what extent 

o comparative data across countries on school and teacher practices in the teaching of 
reading 

o establishment of a 1990 baseline database in 30 countries 

These products provide each education system with data on literacy levels that can be related to 
economic, health, and other educational indicators. (The study identified variables that, if changed. 
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could help to improve literacy rates.) On an international level, the large 1990 database will be 
useful in the future for assessing changes in literacy achievement over time. 

Because the United States has large national and state data collection efforts underway, the U.S. 
national steering conmiiuee for the iEA Reading Literacy Study considers the primary purpose of 
participating in this study to be the gathering of comparative international data and information that 
will not only compare student competence but provide insights to help explain differences in 
pcrtbrmance by U.S. students and other students. 

The IEA Reading Literacy Study provides an opportunity to investigate how reading is taught in a 
large number of education systems with wide variations in traditions, economic development, school 
organization, classroom conditions, teacher characteristics, and orthography, and to contribute to a 
more context-sensitive perspective on successful means of teaching reading and successful reading 
acquisition. Because there is much unity in the culture of reading instruction around the world, texts 
have many universal features in their functions, contents, structures, and textual characteristics. This 
unity makes it possible to translate texts. The Reading Literacy Study assumes that there is a 
conmion basis that makes int^ational comparison possible. 

Organization and Management IEA selected an international coordinator, established an 
international coordinating center and data processing center at the University of Hamburg in 
Germany, and appointed a steering committee and technical advisor. The members of the steering 
conunittee, the international coordinator, and the data manager worked together with the national 
research coordinators who were responsible for the conduct of the project in their respective 
countries. 

The national center in each participating country sqppointed a national research coordinator to assume 
overall responsibility for the project within that country and a national committee composed primarily 
of reading literacy experts. National centers that wished to attach additional research to the project 
sqppointed appropriate additional members to their national committees. National centers s^pointed 
staff as required. 

All conceptual and operational decisions were made cooperatively by the steering conmiittee and 
national research coordinators. The international sampling referee approved and checked sampling 
plans. A data processing team at the University of Hamburg created the computer-stored working 
files. 

The U.S. national center and national research coordinator are both located at the National Center for 
Education Statistics. As subcontractors, Westat, Data Recognition Corporation, and the Council of 
Chief State Scliool Officers have worked with the NCES to conduct the testing, analysis, and 
reporting for the United States. The U.S. national center (NCES) has provided technical assistance to 
other participating country national centers, through two seminars. In the seminars national research 
coordinators, intemationd coordinating renter staff, NCES rq)resentatives, and Westat staff worked 
together to generate appropriate descriptive tables reflecting the core concerns of the international 
Reading Literacy Study. At the conclusion of each seminar, participants had the basis for a national 
report in the form of a set of tabulations around which such a report could be built. 
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Pattieipants 

Research institutes from thirty-two systems of education participated in the study. Twenty- 
seven countries participated in testing populations A and B: Belgium (French), Canada 
(British Columbia), Cyprus, Denmark, Finland, France, Germany (East), Germany (West), 
Greece, Hong Kong, Hungary, Iceland, Ireland, Italy, The Netherlands, New Zealand, 
Norway, Portugal, Singapore, Slovenia, Spain, Sweden, Switzerland, Thailand, Trinidad & 
Tobago, the United States, and Venezuela. One country participated only in Population A: 
Indonesia; four countries participated only in Population B: Botswana, Nigeria, the 
Philippines, and Zimbabwe* 

Population A was defined as: all students attendmg mainstream schools on a full-time basis at 
the grade level in which most students aged 9:00-9:11 years were enrolled during the first 
week of the eighth month of the school year. Population B was defined as: all students 
attending mainstream schools on a full-time basis at the grade level in which most students 
aged 14:00-14:1 1 years were enrolled during the first week of the eighth months of the school 
year. 

Sample The formal survey was conducted on scientifically selected national samples of 9- 
and 14-year-olds, typically 1,500 to 3,000 pupils per country and their teachers. To obtain 
comparable samples of students, multi-stage sampling was used in each country and schools or 
classes were typically drawn with a probability proportional to the size of the school or class. 
Where schools were drawn, an intact class was selected at random within each school, but in 
Population B some national research coordinators selected students at random from all classes 
in the grade level in the school. To overcome fluctuations in the execution of the sampling, 
weighting was used to adjust for any variations in the probability of selecting students. These 
sampling weights were used in all data analyses. 

The U.S. sample included 332 participating schools. The target population is similar to that 
of the National Assessment of Educational Progress in that it uses the most inclusive 
population possible. However, unlike NAEP, lEA Reading Literacy targeted all students in 
the modal grades for 9- and 14-year-olds. This differs from the NAEP target, because lEA 
Reading Literacy tested intact classes and 14- rather than 13-year-olds. 

Procedures and Summan of Content 

Reading Test: The Reading Literacy Study reading test included passages and items r^resenting 
narrative text, expository text, and documents. The tasks required an ability to comprehend specific 
vocabulary and continuous text and to locate and use document information. 

Background Questionnaires: The student questionnaire was designed to obtain information on the 
students' home and school circumstances, and included questions concerning parents' education and 
interaction with the student and his schooling, the students' reading and leisure time activities, and 
personal descriptive data. The teacher questionnaire was designed to obtain information on each 
teacher's background, instructional practices, and beliefs, and included questions concerning training 
and experience, personal descriptive data, classroom materials and environment, and teaching 
strategies and activities. The school questionnaire was designed to obtain information about Uie 
school circumstances and policies as viewed by the principal, and included qu^tions concerning the 
principals' training and experience, personal descriptive data, and school descriptive data. 
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National Case Study Questionnaire: The national case study questionnaires were designed to obtain 
information on national policy, enrollment patterns, and economic conditions and included questions 
on curriculum, age of entry to school, teachers* salaries, hours of instruction, percentage of an age 
group enrolled in formal schooling, and teacher training emphasis. National research coordinators 
completed these questionnaires* 

Data CoUeetion and Analyses All students took reading tests and responded to a background 
questionnaire about their reading at home and at school. Teachers and school principals responded to 
questionnaires about themselves, their teaching, and the school organization. Each national center 
completed a national case study questionnaire. Constructs were identified as the basis of the 
international data analysis. The major analysis was performed for Population A. 

In December 1991 the U.S. national steering committee presented a symposium at the National 
Reading Conference meeting in Palm Springs, California. This was the first presentation of the U.S. 
data with regard to reading test design, family characteristics, differences in scaling, teacher 
characteristics, and principals as instructional leaders. 

Four reports of preliminary international results were published in 1992-1993; the main international 
research report was published in 1994. The lEA Reading Literacy Study international database was 
established in 1993. A full study archive for the study was established in 1994. The archive includes 
the conc^tual framework; test items as submitted and revised; test item selection criteria, national 
test instruments and case study questionnaires; national sample designs; final versions of reading tests 
and questionnaires, manuals, codebooks, and datasets; and national and international reports and 
publications* 

Three U.S. national reports will be published in 1994. The first is more tedmical in nature and 
covers the procedures used in gathering and processing the data for the U.S. portion of the 
international study; it also explores issues of validity, comparisons to the National Assessment of 
Educational Progress, the relationship of the Reading Literacy Study to the research literature, and 
modeling techniques used in developing the U.S. analyses. The second U.S. national rqwrt is 
intended for a more broad general audience. The third consists of papers that deal with issues in 
comparative studies. A fourth report will focus on the United States in international perspective and 
include special analyses of modeling, bilingual instructional programs, quality of school life, and 
instructional practices. 

Timeiable 

1986 The lEA General Assembly approved the Reading Literacy Study and formed a 
steering conmiittee and technical advisory group. 

1988 The international coordinating center was established and the international coordinator 
was selected, both located at the University of Hamburg. 
Participating country research institutes appointed national research coordinators. 
National research coordinators met. (November) 
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1989 The international steering committee met in San Francisco to review and revise test 
items* (April) 

National research coordinators met in Washington to agree on field test items and 

procedures and to sq)prove sampling plans* (June) 

The international steering conmiittee met with national research coordinators. 

(November) 

The U.S. national advisory panel was formed; it met in December. 
1988-1990 

Items for the tests and questionnaires were generated, translated, and pilot tested. 
(November 1988-July 1990) 

1990 The U.S. national steering conmiittee met in Boston. 

The international steering committee met in Hamburg. (May) 
The international steering committee and national research coordinators met in 
Frascati, Italy to reach agreement on fmal instruments and study design. Final tests 
were selected. (July) 

A baseline database was established in 30 countries. 
1990-1991 

The formal survey was conducted. (October 1990-April 1991, depending on the 
school year in each country) 

1991 The international steering committee held a technical advisory committee meeting. 
(April) 

The U.S. national steering committee met to review the U.S. national data and plan 
for analysis and reporting. (June) 

The international steering committee held a national research coordinators meeting in 
Denmark. (October) 

The U.S. national steering conmiittee presented a symposium at the National Reading 
Conference in Palm Springs, California. (December) 

1992 National research coordinators met in Madrid. (October) 
1992-1994 

Preliminary results were published in four books. 

1993 The international coordinating center completed its work and transferred data tapes 
and other materials to lEA headquarters. 

An international database was established. 

1994 A full study archive was established. 
The main research rq)ort was published. 
Four U.S. national reports will be published. 

PublitatiQns 

How in the World Do Students Read? Preliminary results. lEA, The Hague, The 
Netherlands. (1992) 

Effective Schools in Reading: Implications for Educational Planners (An Exploratory Study). 
Preliminary results. lEA, The Hague, The Netherlands. (1992) 
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Teaching Reading Around the World. Preliminary results. lEA, The Hague, The 
Netherlands. (1993) 

Gender Differences in Reading. Preliminary results. lEA, The Hague, The Netherlands. 
(1994) 

International Rqwrt: The lEA Study of Literature: Achievement and Instruction in Thirty- 
two School Systems. Oxford: Pergamon Press. (1993) 

The lEA Reading Literacy Study: Technical Report. lEA, The Hague, The Netherlands. 
(1993) 

Main research report. Oxford: Pergamon Press. (1994) 
U.S. national r^rts will be published in 1994. 

Database 

The lEA Reading Literacy Study international database was established in 1993. 

Funding International funding was provided by The MacArthur Foundation, The Maxwell Family 
Foundation, The Melon Foundation, the National Center for Education Statistics (through the National 
Academy of Sciences), Commission of the European Communities, UNESCO, and annual 
contributions from participating countries. Participating country research institutions also provided 
funds for their national costs of conducting the study and for travel and subsistence costs for their 
national research coordinators to attend national research coordinator meetings. 

In formation Sources 

Marilyn Binkley, National Research Coordinator 
National Center for Education Statistics 
555 New Jersey Ave., N.W. 
Washington, D.C. 20208 

telephone: 202/ 219-2195 

facsimile: 202/ 219-1736 

e-mail: bmt@cu.nih.gov 

International Association for the Evaluation 
of Educational Achievement Secretariat 
c/o SVO 

Sweelinckplein 14 

2517 GK The Hague THE NETHERLANDS 
telephone: 3 1-70-346-96-79 
facsimile: 31-70-360-99-51 
e-mail: iea®svo.nl 

Gordon M. Ambach, U.S. Liaison to lEA 
Council of Chief State School Officers 
One Massachusetts Ave., N.W.» Suite 700 
Washington, D.C. 20001-1431 

telephone: 202/ 408 =5505 

facsimile: 202/ 408-8076 

e-mail: gambach@nas.edu 
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Binkley» Marilyn R. 

1989 lEA Reading Literacy Study. SUtus reports prepared foi the Board on International 
Comparative Studies in Education* January and August. Presentations to the board. April 
and September. 

1990 lEA Reading Literacy Study. Status reports prquued for the Board on International 
Comparative Studies in Education. April and September. 

1991 lEA Reading Literacy Study. Status reports prepared for the Board on International 
Comparative Studies in Education. Janv3ary> April, and September. 

1991 lEA Reading Literacy Study: dissemination plan. Presentation to the Board on International 
Con^Murative Studies in Education. May. 

1992 lEA Reading Literacy Study. Status tepotts prepared for the Board on International 
Comparative Studies in Education. January and April. 

1993 TEA Reading Literacy Study. Status reports prepared for the Board <fn International 
Coiiq[>arative Studies in Education. June and October. 

Binkley» Marilyn R., and Gary Phillips 

1988 lEA Reading Literacy Study: pkms for the U.S. national study. Presentation to the Board on 
Intematicmal Comparative Studies in Education. November. 

1989 lEA Reading Literacy Study. SUtus report pr^red for the Board on International 
Conq>arative Studies in Education. April. 

Binkley» Marilyn R., and Trevor Williams 

1992 lEA Reading Literacy Study: status rqx)rt. Presentation to the Board on International 
Conq>arative Studies in Education. October. 



Elley, Warwick B. 

1988 lEA Reading Literacy Study: International a^>ects of the study. Presentation to the Board on 

Intematioiud Comparative Studies in Education. November. 
1992 How in the world do students read? The International Association for the Evaluation of 

Educational Achievement^ The Hague, The Netherlands. 
1994 [title?] Main research rqport of the lEA Reading Literacy Study. Oxford: Pergamon Press. 



Eliey, Wanvick N., Editor 

1993 International Report: The lEA Study of Literature: Achievement and Instruction in Thirty- 
two School Systems. Oxford: Pergamon Press. 

International Association for the Evaluation of Educational Achievement 

1993 Activities, Institutions, and People: lEA Guidebook 1993-1994. IEa, The Hague, The 
Netherlands. 

1993 lEA Reading Literacy Study: international daUbase. lEA, The Hague, The Netherlands. 

1993 The lEA Reading Literacy Study: Technical Report. lEA, The Hague, The Netherlands. 

1994 lEA Reading Literacy Study: full study archive. lEA, The Hague, The Netherlands. 

Lundberg, Ingvar, and Pirjo Liniukyla 

1993 Teaching Reading Around the World: lEA Reading Literacy Study. January. The 

International Association for the Evaluation of Educational Achievement, The Hague, The 
Netherlands. 



Postlethwaite, T. Neville, and Kenneth N. Ross 

1992 Effective Schools in Reading: Implications for Educational Planners: An Exploratory Study. 
November. The International Association for the Evaluation of Educational Achievement, The 
Hague, The Netherlands. 

Wagemaker, H., Georgia Polydorides, and Michael Martin, Editors 

1994 Gender Differences in Reading. lEA, The Hague, The Netheriands. 
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NOTE: This study summary was reviewed and edited by Marilyn Binkley at the National Center for 
Education Statistics in Washington, D.C. on June 22, 1994. 
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STUDY ON PERFORMANCE STANDARDS IN EDUCATION 



Intematiomd Orfanization U.S. Dq)artment of Education, Office of the Under Secretary and the 

Organisation for Economic Co-operation and Development 

Years of Data CoUection 1994 

Purpose Through the Goals 2000 education reform legislation, the federal government is supporting 
state efforts to develop or adopt content and performance standards for their school children; it also 
establishes a National Educational Standards and Improvement Assessment Council to certify state- 
submitted standards and assessments. Legislation reauthorizing the largest federal elementary and 
secondary education programs requires as a condition of federal funding that U.S. states submit state 
plans describing coherent and challenging content and performance standards that they will establish 
or use for all children, as well as what the states will do to enable children to meet these standards. 

Because the potential influence of such standards is so great, and because the complexity of 
implementing them in our diverse educational systems is so challenging, U.S. policy makers seek to 
learn throu^ cooperation with other countries that already have substantial experience in developing 
and implementing education standards, particularly the members of the Organisation for Economic 
Co-operation and Development. In order for this international experience to have an impact on the 
standards being developed in the United States, it must be compiled, analyzed, and disseminated 
before subject-area standards are finalized and disseminated in the United States. 

In order to obtain the required information and analyses on education standards in other countries, the 
U.S. Dq>artment has contracted an agreement with the Organisation for Economic Co-operation and 
Development. In carrying out this agreement, a series of approximately eleven discrete analytical 
reports will address the question: What can the United States learn through interaction with member 
countries of the OECD sboui how to set and implement subject-matter standards (especially 
performance standards)? 

Through this study the OECD is developing the following products: 

o A series of approximately eleven brief written analyses (ten country specific and one 
cross-national) resulting from the analyses on education standards. 

o An annotated compilation of standards-based assessment materials from selected 
OECD countries (as sq>proptiate, these may be in the form of appendices to the 
country specific or cross-national written analyses). 

o Brief written records of meetings between officials and educators from the 
participating countries and the OECD. 

OrtaniTUtilt n Management In accordance with OECD work underway in 1993 on school 
effectiveness, the OECD is responsible for the following for this study: coordinating the work of 
obuining the required information and analyses on education standards; providing its expertise, staff, 
and consultant time to link expert researchers and policy makers from OECD countries with their 
counterparts in the United States; and contracting with expert consultants to conduct the analyses on 
education standards, and carry out other specific analyses. 
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The U.S. Dq)artment of Education Office of the Undersecretary is responsible for funding the OECD 
for its activities for this study; designating a project officer to serve as a single point of contact for 
liaison with the OECD; reviewing, providing comments, and zpptoving all statements of work, 
proposed consultants, draft rq)orts, and schedules for completion of specific subtasks; and consulting 
and coordinating with other offices within the Dq>artment of Education and other federal government 
agencies that have significant ongoing activities in the area of education standards (for example, the 
Office of Educational Research and Improvement and the National Academy of Sciences). 

Design 

Participants 

Australia, Canada (Ontario), England and Scotland, France, Germany, Ireland, Japan, 
Spain, Sweden, and the United States 

Smpk 

Not applicable. 

Procedures and Summan o f Content Country studies: Written essays may emphasize distinctive 
features in a country's system and whether the writer considers them as strengths or weaknesses. 
They will address: 

o reasons for concern about standards 
o differences in student achievement 
o setting and monitoring standards 

o procedures and methods (who sets the standards; how are they managed and financed, 
how are consistency and fairness achieved, what factors influence standard levels, 
what are the mechanics of the standards setting process, what constitutes a 
performance standard? how are performance standards applied, how and when are 
standards reviewed, how is attainment of performance standards determined, how are 
instruments constructed and scored, how are attainment and analyses reported, and are 
there provisions for testing and scoring children with special needs?) 

o preconditions and problems 

0 policy formulation and decision-making 

o the international dimension (what is the value of comparing procedures in OECD 
member countries for setting performance standards?) 

Data Collection and Analyses Analytical activities being used are: 

1. Country-specific technical studies of the development and implementation of subject- 
area standards and standards-based assessments, with special emphasis on performance 
standards (i.e.» the levels at which students are expected to perform). The specific 
countries being analyzed were selected by mutual agreement of the OECD, the U.S. 
Department of Education, and the countries themselves. The studies are based on 
interviews and discussions with relevant officials and analyses of primary and 
secondary data sources. 
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Of particular interest in designing and carrying out these studies are: 

o the policies, procedures, criteria, and threshold(s) set for passage and how and 

by whom they are determined; 
o techniques of mediation of scores or ratings, arbitration and review 

procedures, and the outcomes of these techniques; 
o procedures for students to meet the standards; 
o the use, methodology, and success of "performance-based assessment;** 
o whether and how performance standards are or have been used in OECD 

countries to assess educational institutions and local-government education 

agencies, for purposes of continuous improvement and accountability* 

2« Comparative studies that synthesize and analyze the information gathered under the 
country-specific work carried out under the country-specific technical studies* 



Timffdbk 

1994 Country reports will be in final draft. (Fall) 
Synthesis paper will be prepared* (Fall) 

Publications 

To be determined* 
Database No data base is available to the public at this time* 
Funding Funding is provided by the U.S* Department of Education. 



In formation Sources 

Marshall S* Smith, Undersecretary 
U*S* Department of Education 
400 Maryland avenue, S*W*, Room 3127 
Washington, D*C. 20202 

telephone: 202/ 401-3132 

facsimile: 202/ 401-3036 

R*A* Cornell, Deputy Secretary General 

Organisation for Economic Co-operation and Development 

2, rue Andre-Pascal 

75775 Paris Cedex 16 FRANCE 

telephone: 33-1-45-24-82-00 

facsimile: 33-1-45-24-85-00 
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U.S. Department of Education and Organisation for Economic Co-operation and Development 

1993 Agreement Between the U.S. DepBimcnt of Education Office of the Undersecretary and The 
Organisation for Economic CM>peration and Development. U*S. DqMUtment of Education, 
Washington, D.C. and OECD, Paris, France. 

U.S./OECD Study on Performance Standards in Education: Quality, Curriculum, Standards, 
Assessment: Guidelines and Questions for Country Studies. U.S. Department of Education, 
Washington, DX* and OECD, Paris, France. 



♦♦♦♦♦♦ 

NOTE: This study summary was reviewed and edited by Lenore Garcia at the U.S. Department of 
Education Office of Planning Services in Washington, D.C. on June 8, 1994; Lenore Garcia provided 
database information on February 17, 1995. 
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SURVEY OF MATHEMATICS AND SCIENCE OPPORTUNITIES (SMSO) 



International OrfcniTOtion Michigan State University 

Years of Data CoUeetion 199M992 

1992-1993 

Purpose The purpose of the Survey of Mathematics and Science Opportunities is to develop better 
opportunity-to-leam methodology for the lEA Third International Mathematics and Science Study and 
to characterize the educational opportunities for learning mathematics and science in SMSO 
participating countries. This survey is a research and development effort studying intended 
curriculum, opportunity to learn, classroom processes in mathematics and science instruction, and 
teachers' knowledge of mathematics and science; it will provide conc^tual, empirical, and 
instrumentational input for TIMSS. The SMSO survey will enable TIMSS to provide better 
characterizations of achievement level differences among and within countries and to explain the 
curricular context from which these differences arise. 

< ?ry g ff ? >^^» Management The SMSO project and project director are located at the Michigan 
State University College of Education. The project director works with a team of specialists and an 
advisory board of internationally known experts. 

Design To develop measurement of intended curriculum, opportunity to learn, classroom processes, 
and teachers* knowledge of mathematics and science, and to provide a characterization of 
mathematical and science opportunities in each participating SMSO country, the following were used: 

0 Analysis of curricular materials and development of a curriculum framework onto 

which the intended curriculum of a country could be mapped. These are multi-aspect 
frameworks that include a content domain, performance expectations, and context. 
The curriculum framework was then applied to a set of curriculum syllabi and revised 
as necessary. 

0 Classroom-based study in each of the SMSO participating countries. As separate 
aspects of the classroom studies, the development of improved measures of subject 
matter-specific pedagogy, teacher knowledge, and opportunity to learn were 
combinai. 

Participants 

France, Japan, Norway, Spain, Switzerland, and the United States 

Sample The materials prepared to measure subject matter specific pedagogy, teacher 
knowledge, and opportunity to learn were used in a nationally generated sample of 20-30 
classrooms within each of tfie participating countries for a full school year. 

Procedures and Sl larv o f Content Logs, case studies, and other instruments were used in the 
selected classroom' . Trained interviewers visited the schools every other week to collect logs and 
other materials responded to by teachers. Teachers were also interviewed and classroom observations 
of mathematics and science lessons were conducted. 
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Data Collection and Analyses During 1992-93 the focus was on self report measures validated in the 
case studies. Analysis of the data was initiated witli a meeting of rq>resentatives from each 
participating count^. Rq)resentatives compiled the data» documented any irregularities in the data 
collection procedures, and established data analysis plans* 

SMSO has developed curriculum frameworks for mathematics and science. The frameworks are 
systems for coding and describing materials such as curriculum guides, syllabi, textbooks, and 
examinations along each of several dimensions and with regard to multiple attributes. SMSO has 
contributed to TIMSS by developing various teacher and school questionnaires, the TIMSS 
Conc^tual Framework for Contextual Questions, the TIMSS Mathematics and Science Curriculum 
Frameworks, the TIMSS Document Analysis Manual, and the TIMSS Topic Trace Mapping 
Instruaions and Forms. 

Six countries have participated in the development of a conceptual model of educational opportunity 
and in the development and piloting of related questionnaires and procedures in preparation for 
TIMSS. During 1991-92 observations were made in multiple classrooms in France, Js^an, Norway, 
Spain, Switzerland, and the United States. Discussions of these observations served to inform 
questionnaire development for TIMSS. 

The Opportunity to Learn pilot was conducted in over 75 classrooms in France, Japan, Spain, 
Switzerland, and the United States. It gathered information from teachers and students on students' 
opportunity to learn various topics as represented in the TIMSS curriculum frameworks. This pilot 
was designed to address issues of validation of teachers' indications of students' opportunity to learn, 
which can provide a rich and valuable context for understanding students' performance on 
achievement assessment measures. 

A report on what SMSO research has learned was presented at a symposium at the American 
Educational Research Association annual meeting in Atlanta. 

In January 1994 representatives from the six SMSO countries held an international working meeting, 
focusing on classroom observations of teachers' instructional practices. Participants read, discussed, 
and analyzed summaries of over 100 classroom observations that had been conducted in the SMSO 
countries. Analyses of these observational data sununaries will be presented in a book on 
instructional practices. 

1991 Development of curriculum frameworks began. 

A taxonomy of frameworks was drafted and used in trial coding. 
Development of methodology and instruments for classroom studies began. 
Data was reviewed and taxonomy revised. 
A protocol manual was developed. 

1992 The full-scale study was conducted. 
Curricular materials data were analyzed. 
Curricular materials report was written. 

A trial of test items was conducted using students in classroom study. 
U.S. data was coded. 

Data from SMSO countries was banked, cleaned, and analyzed. 
A final instrumentation reconunendation was made to TIMSS. 
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1992-'1993 

Classroom observations were made in the 6 SMSO countries. 

1992 Three international meetings were held to discuss classroom observation information. 

1993 A report on what SMSO research has learned was presented at a symposium at the 
AERA annual meeting in Atlanta. (April) 

A meeting was held to produce versions of the student, schooU and teacher 
questionnaires for international piloting. (May) 

Final pilot forms were submitted to the TIMSS international coordinating center. 
Final pilot forms were used in the October teacher and school questionnaire pilot. 
International meeting was held, to discuss plans for further data analyses and 
publications, (fall) 

Data collection for the Opportunity to Learn pilot was completed. 

1994 Representatives from the six SMSO countries held an international working meeting, 
focusing on classroom observations of teachers' instructional practices. Paris. 
(January) 

A revised version of the student questionnaire was used in the March international 
field trial. 

Two major SMSO papers were published by the TIMSS international coordinating 
center 

Rq>resentatives from the six SMSO countries held an international working meetings 
to complete analyses for the Opportunity to Learn pilot and discuss drafts of the 
Instructional Practices book, (sununer) 

The Instructional Practices book will be completed by the end of 1994. 

Two SMSO papers were published by the TIMSS international coordinating center. 

Publications 

Two major SMSO pj^ers have been issued as lEA Third International Mathematics and 
Science Study international coordinating center publications: 

Test Blueprints: A Description of the TIMSS Achievement Test Content Design. University 
of British Columbia, Vancouver, Canada- (1994) 

TIMSS Educational Opportunity Model: Detailed Instrumentation And Indices Development. 
University of British Columbia, Vancouver, Canada. (1994) 

The SMSO Research Report Series includes 57 draft and final documents on topics such as 
mathematics, science, and opportunity-to-leam curriculum framework, and topic trace 
mapping. It also includes questionnaires and manuals. 

Three proposed SMSO survey reports are: 

Instructional practices: An international report of differences across 6 countries (Japan, 
France, Norway, Switzerland, Spain, United States) In the delivery of classroom instruction 

Opportunity-to-leam achievement study: The report of a study of opportunity to learn - how 
it Is measured. Its validity, and its reliability 
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Qualitative textbook analyses: A qualitative description of mathematics textbooks in the six 
SMSO countries 



At this time the information that has been gathered for the SMSO/Third International 
Mathematics and Science Study is not public information. It will be a year or two before this 
information can be released. 

Funding Funding is provided by the National Science Foundation in conjunction with the National 
Center for Education Statistics. 

Information Sources 

William H. Schmidt, Project Director 
463Erickson Hall 

Michigan State University College of Education 
East Lansing, Michigan 48824*1034 

telephone: 517/ 353-7755 

facsimile: 517/ 432-1727 

e-mail: wschmidt@msu.edu 



McKnight, Curtis, William H. Schmidt, and Senta Raizen 

1994 Test Blueprints: A Description of the TIMSS Achievement Test Content Design. University 
of British Columbia, Vancouver, Canada. 

Schmidt, William H. 

1991 Survey of Mathematics and Science Opportunity and U.S. TIMSS - sUtus of activities. 
Presentation to the Board on International Comparative Studies in Education. September. 

1992 Survey of Mathematics and Science Opportunity and U.S. TIMSS - sUtus of U.S. 
participation. Presentation to the Board on International Comparative Studies in Education. 
February. 

1992 Survey of Mathematics and Science Opportunity and U.S. TIMSS - sUtus of activities. 
PresenUtion to the Board on International Comparative Studies in Education. October. 

1993 Survey of Mathematics and Science Opportunity: Results of the curriculum analysis. 
Presentation to the Board on International Comparative Studies in Education. June. 

1994 TIMSS Educational Opportunity Model: Detailed InstrumenUtion And Indices Development. 
University of British Columbia, Vancouver, Canada. 

1994 TIMSS curriculum analysis plans. PresenUticm to the Board on International Comparative 
Studies in Education. May. 

Schmidt, William H., and Senta Raizen 

1991 Survey of Mathematics and Science Opportunity - update. PresenUtion to the Board on 
International Comparative Studies in Education. May. 

Survey of Mathematics and Science Opportunities 



1992 TIMSS Mathematics Curriculum Framework. March. SMSO, Michigan State University, 

East Lansing, Michigan. 
1992 TIMSS Document Analysis Manual. March. SMSO, Michigan SUte University, East 
Lansbg, Michigan. 

1992 TIMSS Topic Trace Mapping Instructions and Forms. SMSO, Michigan State University, 
East Lansing, Michigan. 
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1993 TIMSS: Curriculum Analysis: A Content Analytic Approach. Research R^rt Series 
No. 57. March. SMSO, Michigan State University, East Lansing, Michigan. 



****** 



NOTE: This study summary was reviewed and edited by William Schmidt at Michigan State 
University in East Lansing on July 10, 1994; database information was provided by Schmidt on 
November 29, 1994. 
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THIBD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY (TIMSS) 



International Orfanization International Association for the Evaluation of Educational 

Achievement 

Years of Data CoUection 1993-1995 

1997-2000 

Purpose The Third International Mathematics and Science Study will be a major source of 
information for discussion of the quality of education around the world. It will not only measure 
student achievement in mathematics and science internationally, but will also investigate differences in 
curriculum and instruction, and include alternative assessment options* 

Policy makers are increasingly called upon to make decisions affecting the quality of education in 
their nations* The quality of the educational systems of a nation is seen to be linked^to the ability of a 
nation to compete in the international marketplace* Therefore, increasingly policy decisions for 
education are being examined in light of the economic status of a nation and policy makers are 
looking to the results of international studiu as a source of information* During diis time of rs^id 
and pervasive technological advancement, their interest is particularly focused on the areas of 
mathematics and science achievement that are seen to be strongly linked with economic development. 

TIMSS will simultaneously investigate science and mathematics curricula and achievement* This 
design will enable researchers to collect data on two different subjects concurrendy and will enable 
educators and policy makers to identify relationships between the two disciplines as they are intended 
for learning, as they are taught, and as they are learned* 

Previous lEA studies in the areas of mathematics and science have produced valuable insights into a 
number of aspects of the teaching and learning process* In addition, they have provided a number of 
important lessons with respect to the design and conduct of large-scale international research projects 
in education* TIMSS will build on this foundation, and will provide ftirther information about several 
important matters, such as: 

0 The need to provide current national and international information that educational 
systems can use to compare and contrast their curricula, teaching practices, and 
student outcomes with die educational systems from other countries of interest* 

0 An assessment of the potential impact that alternative curricular offerings, teaching 
strategies, and administrative arrangements have on learning* 

o An identification rf what is possible in the teaching of mathematics and science. For 
example, results from the Second International Mathematics Study showed extremely 
high growth rates in two countries, and that finding indicates that more significant 
growth rates might be possible in other countries* Identification of the underlying 
causes of those high growth rates would have profound educational implications* 

0 A greater understanding of how and why student attitudes change, and what 

relationship the development of positive attitudes bears to classroom practices* Such 
an understanding is crucial to the development of a more complete picture of how 
science and mathematics learning takes place. 
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Educational inputs, processes, and outputs constitute three basic components of the educational 
system, and each of these is related to the others in a complex, integrated whole. An important 
objective of TIMSS is to examine the comparative influence of a variety of constituent variables 
within each component on the success of a given educational system as indicated by measures of 
student achievement, participation, attitudes, and aspirations. 

lEA has established a cycle of studies covering the major areas of the school curriculum: the 1990s 
will be the TIMSS decade. It has been more than 10 years since the lEA Second International 
Mathematics Study and Second International Science Study were conducted (although publication of 
most of the international reports from those studies was delayed because of lack of funding needed to 
carry out the analyses of the international data). In the interim, interest in cross-national comparisons 
generally, and in the areas of science and mathematics specifically, has increased dramatically. 

Mathematics and science are important components of the school curriculum in every country. They 
are seen as integral components of every student's educational program. As the role and impact of 
technology continue to increase in society and in the workplace, the concepts, processes, and skills of 
science and mathematics are likely to become even more highly valued and the importance of these 
subjects in school curricula to become even greater. 

Many countries around the world are currently involved in major reforms of their curricula in 
mathematics and science, and many of them will look to lEA and to TIMSS for the kind of 
information they need to guide that process. The findings from TIMSS will be of great interest 
internationally to educators, curriculum developers, researchers, policy makers, and politicians. 

Some of the major topics on which TIMSS will report are: 

0 Mathematics and science curricula. The results of the curriculum analysis will 
document international variation in mathematics and science curricula. 

0 Opportunity to learn. Opportunity*to-leam data, in conjunction with the curriculum 
analysis, will illustrate what is possible in the teaching of science and mathematics. 

0 Students' achievement. TIMSS will give considerable emphasis to students' 

achievement, including students' ability to apply their knowledge and skills in non- 
routine settings. 

0 Use of technology. The role of technology in the teaching and learning of 

mathematics and science, particularly as regards the use of calculators and computers 
for instructional purposes, is a topic of significant interest for TIMSS. 

0 Participation rates. The participation of students in pre-university courses in 

mathematics and science will be examined, particularly with regard to gender-based 
differences. 

0 Tracking and streaming. There is a considerable amount of international interest in 
the kinds of practices employed by schools and school systems to direct students' 
course selection, including tracking and streaming. Information about such practices 
will be collected at several levels within each participating system. 

0 The role of textbooks. An investigation of the nature, role, and influence of officially 
prescribed textbooks on the teaching of mathematics and science is one of the main 
goals of the curriculum analysis component of TIMSS. 

0 Instructional practices. A comparison of instructional practices, based on teacher self- 
report data, will be a major aspect of the study. 
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The data from this rich array of cross-national comparisons of a large number of widely different 
countries should contribute significantly to the unfolding research agenda and to the needs of policy 
makers, researchers, and educators in many countries. 

Orgd^iUltif fn and Management The International Association for the Evaluation of Educational 
Achievement (lEA) has a chair and a board of directors. The lEA Secretariat coordinates all ongoing 
studies. Individual projects have their own international coordination centers responsible for carrying 
out a given study, and these are backed by an international steering committee and the lEA 
international heaidquarters. The international steering conmiittee chak is located at the Ontario 
Institute for Studies in Education in Toronto, Canada. 

The International Association for the Evaluation of Educational Achievement (lEA) headquarters is 
located in The Hague in The Netherlands. The International Coordinating Center for TIMSS is 
located in the Univeisity of British Columbia Dq[)artment of Mathematics and Science Education in 
Vancouver, Canada. The International Coordinator is now assisted by a Dq)uty International 
Coordinator who has a major administrative responsibility for all aspects of the study. As TIMSS 
coordination has decentralized, a range of study activities has been contracted to institutions in several 
countries around the world. 

The 1993 lEA General Assembly confirmed the new position of Study Director for TIMSS. The 
Study Director and Study Center are located at Boston College. The study director has overall 
direaion of all study activities, including those that the International Coordinating Center continues to 
manage. A National Research Coordinator in each participating country is responsible for the 
conduct of the TIMSS study in that country. (The lEA General Assembly improved a motion to 
discontinue the use of the term National Project Coordinator, and to replace it with National Research 
Coordinator, which has been used in previous lEA studies.) 

Data processing for the pilot study is being conducted at the Institute of Comparative Education at 
The University of Hamburg in Germany; data analysis for the pilot study is being conducted in 
Australia at the Australian Council for Educational Research; questionnaires are being developed at 
Michigan State University in East Laming, Michigan; performance assessment tasks and scoring 
rubrics are being developed at the National Foundation for Educational Research in England. 

The study will draw upon the expertise accumulated across the lEA community during the past thirty 
years of IEA*s existence. The roles of the International Coordinating Center and Study Director are 
to coordinate the development of timelines, instruments, sampling procedures, data entiry procedures, 
analysis methods, translation verification systems, and reports from the data supplied by the 
participating educational systems. To insure the integrity of die study, national centers for the 
participating educational systems will use standardized procedures and systems developed by the 
International Coordinating Center and Study Director. 

The U.S. National Research Center and National Research Coordinator are located at Michigan State 
University College of Education in East Lansing, Michigan. U.S. TIMSS data collection is being 
conducted by Westat as contractor to the National Center for Education Statistics. The Westat 
contract also involves several subcontractors: Educational Testing Service, National Computer 
Systems, Harold W. Stevenson of the University of Michigan, and James W. Stigler of the University 
of California at Los Angeles. Each subcontractor operates a specific portion of the contract under the 
overall direction of Westat. A TIMSS Project Officer at the National Center for Education Statistics 
is the government project officer for all contractor activities. 
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Westat is responsible for U.S. TIMSS production of questionnaires and assessments and for receipt 
and processing of completed instruments for the spring 1994 field test and spring 1995 main data 
collections* Westat will receive support from National Computer Systems in these activities. 
Educational Testing Service will also assist by reviewing instruments and materials md collaborating 
with Westat in data analysis activities. 

U.S. states who desire to compare the achievement of their students to that of other nations 
participating in TIMSS have an opportunity to contract with Westai. to participate as a **mini-nation" 
in the TIMSS data collection. 

U.S. TIMSS case study activities will be directed by Harold Stevenson; videotape observation project 
will be directed by James Stigler. 

Design The design of IIMSS includes a basic set of investigations, and participating educational 
systems will be free to select one or more of the investigations and a number of international options. 
TIMSS will focus on the teaching and learning of mathematics and science at three levels of the 
school system: the grade or level at which most students attain the ages of nine and thirteen and the 
grade or levei at which most students are completing the last year of secondary education. One of the 
studies for the latter population will concentrate on an investigation of the scientific and mathematical 
attainments of students completing secondary school with some degree of specialization in either 
mathematics or science. 

The conceptual model for TIMSS was derived in large part from the models used in earlier lEA 
studies (especially the Second International Mathematics Study and Second International Science 
Study), modified and updated to meet the particular demands of the present study. Fundamental to 
the design of TIMSS is the centrality of curriculum as a variable. Also fundamental to the design of 
the study is the relationship between science and mathematics as separate, yet linked, components of 
that curriculum. 

The TIMSS curriculum frameworks are designed to provide rich descriptions of the intended 
curriculum, ones that can also be used in the development of achievement items. The TIMSS 
frameworks are powerful organizing structures that are botli flexible and sophisticated. The multi* 
aspect, multi-category nature of the frameworks enables the description of traditional as well as 
reform-K)riented curricula, providing dynamic ways of characterizing curricula internationally. The 
development of the TIMSS curriculum frameworks in analyzing curricula will help provide new 
insights into curriculum development internationally. 

Participants About 50 educational systems from all regions of the world and at all levels of 
economic development will participate in TIMSS. Participation from developing nations is 
encouraged by several agencies, lEA headquarters, and the TIMSS International Coordinating 
Center. Efforts are being made to provide appropiriate opportunities for meaningful 
involvement by the educational systems in these nations. 

Samp le 

Achievement test booklets for Populations 1 and 2 were piloted to students in grades three, 
four, seven, and eight. Fifty-two countries piloted the instruments. In the United States the 
pilot study involved twenty-seven schools and 3,000 elementary and middle school students 
from California and Michigan. A concerted effort was made to identify school systems in 
rural, suburban, and urban settings. 
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In the United States the teacher questionnaires Hrere piloted to a convenience sample of 50 
Population 1 teachers and 50 Population 2 teachers. The impropriate version of tfie school 
questionnaire was piloted among 25 principals r^resenting all the populations at which a 
country is participating (in the U.S. - Populations 1,2, and 3). 

Countries provide sampling information to the international study director for review and 
approval. Statistics Canada continually advises national research coordinators; it has prepared 
a third version of the sampling manual. 

Procedures and Summary of Content The major source of data on student achievement in 
mathematics and science for TIMSS will be from achievement surveys containing multiple-choice 
items selected on the basis of their relationship to the mathematics and science curriculum frameworks 
developed for the study. The item pool for the adiievement surveys is constructed so as to include 
items designed to measure cognitive behaviors at all levels, and also contains some free-response 
items, mcluding short answer and extended response, as well as performance tasks. Questionnaires 
designed to elicit information about student and teacher backgrounds will be included for descriptive 
comparisons and to act as controls for multivariate analyses. School and classroom variables, 
including school climate, opportunity to learn, and time on task, will be investigated in a variety of 
relational analyses to be conducted. 

In addition to the main study, two international options have been proposed. One international option 
for participating educational systems will be concerned with an in-depth investigation of students' 
problem-solving strategies. A second option will examine the linkage between teaching practices and 
changes in student achievement and attitudes. 

U.S. TIMSS field test and main study data collections: Westat is responsible for production of 
questionnaires and assessments, receipt and processing of completed instruments for the spring 1994 
field test and spring 1995 main data collections. Westat will receive support from National Computer 
Systems in these activities. Educational Testing Service will also assist by reviewing instruments and 
materials and collaborating with Westat in data analysis activities. 

U.S. TIMSS state-level: States who desire to compare the achievement of their students to that of 
other nations participating in TIMSS have an opportunity to contract with Westat to participate as a 
'^mini-nation'* in the TIMSS data collection. Such states will administer the full TIMSS assessments 
and background questionnaires to a random sample of students in their states. Key features of U.S. 
State TIMSS are: 

0 Assessment scores collected through State TIMSS can be compared reliably to 
international TIMSS scores. 



o Data will include both mathematics and science scores. 

o States will have the choice of assessing their 3rd-4th, 7th-8th, and/or 12th-grade 
students. 

0 States will receive internationally comparable student, teacher, and school data from 
TIMSS background questionnaires. 

o Reports are scheduled to be available in December 1996. 

o States will be responsible for funding their own participation in State TIMSS. 
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The State TIMSS program is now available for participation. Details of the design options and costs 
can be obtained from Westat* 



TIMSS case studies of education policy issues in the United States, Japan, and Germany: Year* 
long case studies in ministries, schools, and homes in three countries will supplement the data from 
the main study about four topics relating to opportunity*to*leam issues: teacher working conditions, 
the implementation of standards, how ability differences are dealt with, and the role of school in 
adolescents' lives. 

U.S. TIMSS videotapes of classroom instruction in the United States, Japan, and Germany: 
Videotapes will be used to provide observations of mathematics instruction in a subsample of 100 
eighth grade classrooms participating in TIMSS. These observations will provide both quantitative 
data and qualitative descriptions of how instruction occurs in these countries. 

Data CoUection and Analyses 

In May 1993 the United States participated in the TIMSS Achievement Item Pilot by piloting student 
test booklets in over 25 schools. More than 3400 students from California and Michigan completed 
booklets (twelve different booklets [six mathematics and six science at Population 1] and twenty 
booklets [ten mathematics and ten science booklets at Population 2]. The U.S. National Center 
coordinated all aspects of the U.S. pilot including recruiting and communicating with school district 
personnel, publishing and disuibuting pilot booklets, and entering and cleaning pilot data. Data was 
sent to the lEA data processing center; preliminary data summaries were forwarded to national 
research coordinators. 

The U.S. National TIMSS/Survey of Mathematics and Science Opportunity Center is directing 
development and production of the context questionnaires. The final pilot forms were submitted to 
the International Coordinating Center for use in the October-December 1993 Teacher and School 
Questionnaire pilot in participating countries; the revised version of the Student Questionnaire was 
used in the March 1994 International Field Trial. The Center was also responsible for writing the 
Pilot Manual, Administration Manual, and Data Code books for the international pilot. The U.S. 
National Center is coordinating all aspects of the pilot and is coordinating revision of the 
questionnaires following the pilot. 

The Technical Advisory Committee met in July 1993 in Melbourne, Australia to discuss the continued 
development of the data analysis plan for TIMSS. 

National research coordinators met in Frascati, Italy in October. Their agenda included discussions 
and presentations on the Populations 1 and 2 item pilot results and plans for the field trial operations. 
The national research coordinators reviewed Subject Matter Advisory Conunittee suggestions for 
Populations 1 and 2 item improvement. Educational Testing Service modified the items and 
assembled the items into booklets for the Field Trial. The TIMSS field test was conducted Februai7 - 
March 1994; in the United States Wesut and its subcontractors were responsible for this activity. 
The data collection materials used included 8 two-part, pencil-and-psqper assessment booklets for each 
population ^provided by the lEA international coordinating center) and the U.S. versions of the 
student and teacher background questionnaires (provided by the U.S. National Research Center). 

The Australian Council for Educational Research, with the aid of Stanford Research Institute 
International and many individuals, worked to develop aq>propriate tests for Population 3. There will 
be a mathematics test, a science test, and a computer literacy test fot the general part of Population 3 
and advanced tests for mathematics and physics specialists. 
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A special working group was established to assist the National Foundation for Educational Research 
in developing performance assessment tasks and scoring rubrics. They met in London in February to 
select the items. 

Performance assessments were conducted March - April 1994: in the United States groups of eight 
students in each of eight schools for Populations 1 and 2. Field test data collection for twelfth-grade 
students was then conducted April - May. It involved pencil-and-paper assessment of 50-60 twelfth 
grade students in each of 20 schools. Statistics Canada will review and evaluate field trial operational 
procedures. 

The procedural forms completed by participants in the field trial activities will be reviewed with the 
objective to determine each participant's ability to provide the necessary information and to determine 
how well current procedures work. 

Data from the teacher and school questionnaire pilot was sent from participating countries and is 
being analyzed. In addition to this quantitative data, large data bases have been created containing the 
free responses of questionnaire respondents and the evaluative comments made by the national 
research coordinators regarding specific items on the questionnaires and the pilot in general. Selected 
National research coordinators met in June 1994 to recommend and implement revisions of the school 
and teacher questionnaires. 

TIMSS assessments and questionnaires and administration procedures will be revised in preparation 
for the main study in spring 1995. 

As of April 1994, the TIMSS curriculum analysis had entered its analysis phase. As of March, 
document analysis data firom over 830 documents from 45 countries had been entered into the data 
base. A comprehensive series of meetings with curriculum specialists and research methodologists 
resulted in the design of a variety of innovative analytical techniques for this component of TIMSS. 
This rq)resents a major breakthrough in the comparative analysis of curricula and represents the first 
in what are expected to be many innovations arising from this ground-breaking component of TIMSS. 
Publication of the analysis in international rq?orts is scheduled for late 1994-early 1995. The focus of 
the first r^rt will be on the provision of educational opportunity through the intended curriculum 
and will closely examine conmionalities and differences in curricula, with particular attention given to 
the empirical description of a world core. 

Field Trial data cleaning and preliminary analysts will be done in Hamburg; scaling will be done at 
the Australian Council for Educational Research. 

An international working group on context questionnaires met in Washington, D.C. in May 1994 to 
refine the school, teacher, and student questionnaires. At this working meeting, they reviewed and 
discussed the questionnaires that had been rearranged and reformatted to improve their clarity at the 
Australian Council for Educational Research. Final reconmiendations were then made regarding the 
questions that would be used in the piloting of the school and teacher questionnaires and revisions 
were made in the student questionnaire. 

In October 1994 an international meeting was held at the U.S. National Research Center to examine 
initial dau analysis resulu and to develop strategies for data analyses and reporting as well as 
additional data collection for the TIMSS curriculum analysis. 

The Technical Advisory Committee met in May 1994 to discuss Population 3 definition and sampling, 
student-teacher linkages, "below the line** definitions, field trial daU analysis, management issues, the 
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handling of missing responses during analysis, operations issues, strategy for main survey item 
selection and performance assessment data analysis. 

Members of the recently established international Free Response Item Coding Committee met in 
April-May for preliminary' discussions on the development of a coding strategy for extended response 
items and performance assessment tasks. In response to concerns about the definition and sampling of 
Population 3, the United States and France co-sponsored a meeting of selected individuals to address 
these issues. A group of selected national research coordinators m^ in early June to revise the 
teacher and school questionnaires. The Subject Matter Advisory Committee will meet in July to 
select candidate items for the main survey test bookie. National research coordinators will meet in 
August 1994 for selection of the Populations 1 and 2 main survey instruments. 

In 1993 and 1994 the U.S. national research coordinator, staff, and team of experts made 
presentations at meetings of the American Educational Research Association, the American 
Association for the Advancement of Science, National Association for Research in Science Teaching, 
Comparative and International Education Society, and National Science Teachers Association, and at 
the Council of Chief State School Officers Conference on Large Scale Assessments on: 

o international curriculum analysis 
o design issues for TIMSS 

o measuring learning opportunities and instructional practices in mathematics and 

science in large-scale surveys 
o development of teacher questionnaires 
o what states can learn from TIMSS 

o development of a model for the international study of learning opportunities 
o use of observational research in the design of items for cross-national measurement of 
lesson structure 

o development and validation of instruments for assessing opportunities to learn 
o empirical analysis of intended curricula for a cross-national study of mathematics and 
science 

o preliminary results from TIMSS 

o international mathematics and science curricula 

o international study of opportunities to learn 

o cross-national measurement of opportunities to learn in mathematics and science 
o world'H:lass standards 

In 1994 the Board on International Comparative Studies in Education organized a session on TIMSS 
for the American Association for the Advancement of Science meeting. Papers were presented and 
discussed on evaluation of international comparative studies, TIMSS design, what is expected to be 
learned from TIMSS, and international mathematics and science curricula. 

Also in 1994 two books produced by the Survey of Mathematics and Science Opportunity at Michigan 
State University were published by the TIMSS International Coordinating Center at the University of 
British Columbia. Test Blueprints: A Description of the TIMSS Achievement Test Content Design 
presents the specifications for the TIMSS sttident test. It details the testing time and number of items 
to be pllocated for specific framework categories and the various question types that will be used. 
TIMSS Educational Opporttinity Model: Detailed Instttimentation and Indices Development describes 
how ttie data from the context questionnaires will be used to address specific research questions. It 
also di^ils some proposed analyses and the development of key indices and typologies. 
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1991 The Sampling and Methodology Conunittee met in Vancouver. (June) 
National research coordinators met in Vancouver. (August) 

The international steering conunittee met in Washington, D.C» (May) 

The working group on alternative assessment met at the University of British 

Columbia* (May) 

The Australian Council for Educational Research hosted a conference on assessment 
in mathematics at the elementary, secondary, and pre-tertiary levels. (November) 
Mathematics and science curriculum frameworks were developed. 
Preliminary topic trace and document analysis data of mathematics and science 
curricula were obtained from 30 countries* 
Methodology for the study was developed. 

1992 The Subject Matter Advisory Conmiittee met in Vancouver. (January) 
National research coordinators met. (March) 

Curriculum frameworks were distributed to participating educational systems. 

Student assessment items were collected. 

Document analysis and topic trace coding were conducted. 

1993 The Subject Matter Advisory Committee met in Vancouver. (January) 
The International Steering Committee met in Vancouver. (February) 
National research coordinators met in Vancouver. (March) 

The U.S. National Center/SMSO directed development and production of context 
questionnaires. Final pilot forms were used in the October international Teacher and 
School Questionnaire Pilot. 
The field test was conducted. (February-May) 

An international working group on context questionnaires met in Washington, D.C. 
(May) 

The Technical Advisory Committee met in Melbourne, Australia. (July) 
The U.S. Steering Committee met twice in Washington, D.C. to discuss performance 
testing, achievement test, sampling, and preliminary curriculum analysis data. (May 
and November) 

National research coordinators met in Frascati, Italy. (October) 

Field trials of instruments for all populations were conducted in the southern 

hemisphere. 

Presentations were made by the U.S. national research coordinator and others at 
AERA, CIES, NARST meetings and a CCSSO conference. 



1994 A special working group established to assist NFER in developing performance 
assessment tasks and scoring rubrics met in London. (February) 
A Field Trial data analysis plan was drafted. (March) 

The revised version of the Student Questionnaire was used in the April International 

Field Trial in the northern hemisphere. Data will be submitted to the University of 

Hamburg in June. 

Three books were published. 

Document analysis data was collected and analyzed. 

U.S. Phase 1 quality control activities were completed: data was cleaned and 

assembled for the international curriculum analysis. 

An international meeting was held at the U.S. National Research Center. (October) 
The Technical Advisory Committee met in Boston. (May) 
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The Free Response Item Coding Commit.ee met in Boston. (April-May) 

The United States and France co-sponsored a meeting in Washington, D.C., to discuss 

Population 3 definition and sampling* (May) 

Selected national research coordinators met in Hamburg to revise teacher and school 
questionnaires* (June) 

The Subject Matter Advisory Committee will meet to select candidate items for main 
survey test booklets* (July) 

National research coordinators will meet in Boston for sdection of Populations 1 and 

2 main survey instruments. (August) 

U.S* State TIMSS is available for participation. 

Presentations were made by the U*S* national research coordinator and others at 
AERA, AAAS, NSTA, and NARST meetings and at a CCSSO conference. 
The Board on International Comparative Sttidies in Education organized a session on 
TIMSS for the AAAS meeting* (February) 

Phase 1 msun data will be collected for all populations in the southern hemisphere* 
The TIMSS Encyclopedia (1st edition) will be published, to serve as a reference text 
for all aspects of the study* 

The document analysis international report will be published. 
1994-1995 

International reports will be published on curriculum analysis* 

1995 Phase 1 main data collection will be conducted in the northern hemisphere* (spriiag) 
The TIMSS Encyclopedia (2nd edition with updated information) will be published. 

1996 Phase 1 main data will be cleaned; data banks will be created* 
Reports of U*S. State TIMSS are scheduled to be available. (December) 
Phase 1 main data collection international report will be published. 
Phase 2 will be designed and developed* 

1997-2000 

Phase 2 main data will be collected* 

Phase 2 main data will be analyzed and reported. 

Publications 

TIMSS curriculum analysis: Topic trace mapping. In Prospects. By William H* Schmidt. 
UNESCO. 

Curriculum Frameworks for Mathematics and Science. TIMSS Monograph No* 1. Pacific 
Educational Press, Vancouver, Canada* (1993) 

Test Bluq?rints: A Description of the TIMSS Achievement Test Content Design. University 
of British Columbia, Vancouver, Canada* (1994) ^ 

TIMSS Educational Opportunity Model: Detailed Instrumentation And Indices Development. 
University of British Columbia, Vancouver, Canada. (1994) 

TIMSS Encyclopedia (1st edition, and 2nd edition with updated information) will be 
published* 
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A document analysis international rq>ort will be published. 

The first TIMSS r^rts dealing with international comparisons of student outcomes are 
scheduled for publication in late 1996. International and national r^rts are proposed on 
U.S. and international pre results, curriculum, Survey of Mathematics and Science 
Opportunity, and survey results. 

A Phase 1 main data collection international xepoti will be published. 

At this time, the information that has been gathered for TIMSS is not public information. It 
will be a year or two before this information can be released. 

Funding Funding is provided by the National Science Foundation, in cooperation with the National 
Center for Education Statistics, for the U.S. Na. onal Research Center at Michigan State University. 
U.S. states that participate in state TIMSS will be responsible funding their own participation. 

I nformatwn Sources 

Albert Beaton, Study Director 

Center for the Study of Testing, Boston College 

Campion Hall, Room 320 

140 Conmionwealth Ave. 

Chestnut Hill, Massachusetts 02167*3807 

telephone: 617/ 552-4521 

facsimile: 617/ 552-8419 

e-mail: beatonal®hermes.bc.edu 

David F. Robitaille, International Coordinator 

University of British Columbia Department of Mathematics and Science Education 
2125 Main Mall 

Vancouver, British Columbia, CANADA V6T 1Z4 
telephone: 604/ 822-5337 
facsimile: 604/ 822-4714 
e-mail: David_Robitaille@mtsg.ubc.ca 

Robert Garden, Deputy International Coordinator 
University of British Columbia Faculty of Education 
2125 Main Mall 

Vancouver, British Columbia CANADA V6T 1Z4 
telephone: 604/ 822-9136 
facsimile: 604/ 822-8571 
e-mail: npc®timss.ubc.ca 

International Coordinating Center 

University of British Columbia Faculty of Education 

Vancouver, British Columbia, CANADA V6T 1Z4 

telephone: 604/ 822-9136 

facsimile: 604/ 822-8571 

e-mail: TILMSSJCC<[(mtsg.ubc.ca 
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William H. Schmidt, National Research Coordinator 
463 Erickson Hall 

Michigan State University College of Education 
East Lansing, Michigan 48824-1034 

telephone: 517/ 353-7755 

facsimile: 517 432-1727 

e-mail: wschmidt®msu.edu 

Lois Peak, U.S. TIMSS Project Officer (contract activities) 
National Center for Education Statistics 
555 New Jersey Avenue, Room 304-C 
Washington, D.C. 20208 

telephone: 202/ 219-2195 

facsimile: 202/ 219-1736 

Sandra Reider 

Westat, Inc. 

1650 Research Blvd. 

Rockville, Maryland 208S0 

telephone: 301/ 738-3616 

facsimile: 301/ 294-2038 
(details of U.S. State TIMSS design options and costs; mailing list; periodic updates on the 
status of State TIMSS) 

International Association for the Evaluation of Educational Achievement (lEA) Secretariat 
do SVO 

Sweelinckplein 14 

2517 GK The Hague THE NETHERLANDS 
telephone: 31-70-346-96-79 
facsimile: 31-70-360-99-51 
e-mail: iea@svo.nl 

Gordon M. Ambach, U.S. Liaison to lEA 
Council of Chief State School Officers 
One Massachusetts Ave., N.W., Suite 700 
Washington, D.C. 20001-1431 

telephone: 202/ 408-5505 

facsimile: 202/ 408-8076 

e-mail: gambach®nas.edu 



Ambach, Gordon, Senta Raizen, William Schmidt, Jeamie Griffith, and Lany Suter 

1994 What Can States Learn from the Third International Mathematics and Science Study? A panel 
chaired by Gordon Ambach at the Council of Chief State School Officers National Conference 
on Large Scale Assessment. June. 

Anderson, Barry 

1993 lEA Third International Mathematics and Science Study: Issues related to the sample design. 
Presentation to the Board on International Comparative Studies in Education. 
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Beaton, Albert 

1993 lEA Third intenuUional Mathematics and Science Study. PresenUtion to the Board on 
Intemational Comparative Studies in Education. November* 

1994 lEA Third International Mathematics and Science Study: Current issues. Progress report and 
presentation to the Board on Intemational Conq>arative Studies in Education. January; 
February. 

Beaton, Albert, and Maryellen Harmon 

1994 lEA Third Intemational Mathematics and Science Study: Issues (plans for test development; 
Population 3 meeting; dissemination plans; performance assessment). Status report and 
presentation to the Board on Intemational Comparative Studies in Education* May. 

Beaton, Albert, and Andreas Schleicher 

1993 lEA Third Intemational Mathematics and Science Study: Meeting on TIMSS sample design. 
Presentation to the Board on Intemational Comparative Studies in Education. January. 

Caldwell, Nancy 

1994 lEA Third Intemational Mathematics and Science Study: U.S. TIMSS studies - case studies. 
PresenUtion to the Board on International Comparative Studies in Education. February. 

Cogan, Leland 

1993 Development and validation of instmments for assessmg opportunities to leam. PresenUtion to 
the Comparative and Intemational Education Society regional meeting. October. 

Gilford, Dorothy M., Daniel G. Horvitz, Ronald K. Hambleton, Albert E. Beaton, David E. Wiley, Richard 
Wolfe, William H* Schmidt, and Donald B. Rubin. 

1994 The Third Intemational Mathenutics and Science Study. Session organized by Gilford and 
Horvitz for the Board on Intemational Comparative Studies in Education, with presenUtions by 
Hambleton (Evaluation of Intemational Comparative Studies), Beaton (TIMSS Design), Wiley 
and Wolfe (What is expected to be leamed firom TIMSS), and Schmidt (Intemational 
Mathenuitics and Science Curricula), and discussion by Rubin at the American Association for 
the Advancement of Science meeting. February. 

Griffith, Jeanne 

1991 IE A Third Intemational Mathematics and Science Study: Report of the National Center for 
Education Sutistics Intemational Conference on TIMSS. September. 

Jeanne Griffith, Eugene Owen, and Larry Suter 

1993 lEA Third Intemational Mathematics and Science Study: Issues* PresenUtion to the Board on 
Intemational Comparative Studies in Education. November. 

Horvitz, B.j::1c1 

1993 lEA Third Intemational Mathematics and Science Study: Rqwrt of the board working group 
on TIMSS. PresenUtions to the Board on Intemational Comparative Studies in Education. 
January md June* 

Houang, Richard, Gilbert Valverde, and Virginia Keen 

1993 An empirical analysis of intended curricula for a cross-national study of mathematics and 

science* A fomm for discussion of preliminary resulte of the curriculum analysis component 
of TIMSS presented at the Comparative and Intemational Education Society regional meeting. 
October* 

Intemational Association for the Evaluation of Educational Achievement (lEA) 

1993 Activities, Institutions, and People: lEA Guidebook 1993-1994* lEA, The Hague, The 
Netherlands. 
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lEA TIMSS International Coordinating Center 

1991 TIMSS UPDATE. June 1 and July L lEA TIMSS International Coordinating Center* 
University of British Columbia* Vancouver* Canada. 

1991 TIMSS Quarterly. Summer and Fall. lEA TIMSS International Coordinating Center, 
University of British Columbia* Vancouver* Canada. 

1992 TIMSS UPDATE. December 8. lEA TIMSS International Coordinating Center* University of 
British Columbia* Vancouver* Canada. 

1993 TIMSS Quarterly. October* February 3 and 19** March 19* April 8 and 23* May 7 and 24* 
July 2 and 23. IE A TIMSS International Coordinating Center* University of British Columbia, 
Vancouver* Canada. 

1994 TIMSS UPDATE. January 12 and 28. lEA TIMSS International Coordinating Center, 
University of British Columbia* Vancouver* Canada. 

lEA TIMSS International Study Director 

1994 TIMSS UPDATE. April 22. lEA TIMSS International Study Director* Boston College, 
Chestnut Hill* Massachusetts. 

lEA TIMSS U.S. National Research Center 

1993 lEA TIMSS U.S. National Research Center Report No. 2. August. TIMSS U.S. National 
Research Center* Michigan State University College of Education* East Lansing* Michigan. 

1994 lEA TIMSS U.S. National Research Center Report No. 3. March. TIMSS U.S. National 
Research Center* Michigan SUte University College of Education, East Lansing, Michigan. 

Levine, Daniel 

1992 lEA Third International Mathematics and Science Study: International Coordinating Center 
management of TIMSS operations. Presentation to the Board on International Con^arative 
Studies in Education. May. 

Levine, Daniel* Eugene Owen* and Trevor Williams 

1992 lEA Third International Mathematics and Science Study: Report of management review of 
TIMSS. Presentation to the Board on International Comparative Studies in Education. 
Febniary. 

McKnight* Curtis* William H. Schmidt, and Senta Raizen 

1994 Test Bluq)rints: A Description of the TIMSS Achievement Test Content Design. University 
of British Columbia, Vancouver, Canada. 

McLean* Leslie 

1991 lEA Third International Mathematics and Science Study: Report from the International 
Steering Committee. PresenUtion to the Board on International Comparative Studies in 
Education. May. 

Owen* Eugene* and Lois Peak 

1992 lEA Third International Mathematics and Science Study: Comments on National Project 
Coordinators meeting. Presentation to the Board on International Comparative Studies in 
Education. May. 

Owen* Eugene* Lois Peak* and Larry Suter 

1993 lEA Third International Mathematics and Science Study: Meeting reports (Steering 
Committee* Subject Matter Advisory Committee* Technical Advisory Committee* National 
Research Coordinators). Presentation to the Board on International Con^Nmtive Studies in 
Education. November. 
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Peak, Lois 

1991 lEA Third International Mathematics and Science Study: Plans for case studies and finance 
data collection in connection with TIMSS* Presentation to the Board on International 
Comparative Studies in Education. Sq>tember. 

1994 lEA Third International Mathematics and Science Study: U.S. TIMSS Studies - videoti^ 

studies; field test status tepoti; state participation. Presentations to the Board on International 
Comparative Studies in Education* February and May. 

Plomp, Tjeerd 

1993 lEA Third International MathenuUics and Science Study: SUtus. PrcsenUtion to the Board on 
International Con^Murative Studies in Education. January. 

Raizen, Senta 

1992 lEA Third International Mathematics and Science Study: R<qport of the Subject Matter 
Advisory Committee. Presentation to the Board on International Con^>arative Studies in 
Education. October. 

Raizen, SenU, Lee Burstein, William Schmidt, Smithson, Rolf Blank, and Gilbert Valverde 

1993 Measuring Learning Opportunities and Instructional Practices in Mathematics and Science in 
Large-Scale Surveys: Examining the State of the Art. Session moderated by Raizen at the 
American Educational Research Association annual meeting. April. 

Raizen, SenU, William Schmidt, Ted Britton, Curtis McKnight, Gilbert Valverde, and David Robitaille 
1993 International Curriculum Analysis: A New Approach. Session chaired by Raizen at the 
American Association of Educational Research annual meeting. April. 

Robitaille, David F. 

1992 lEA Third International Mathematics and Science Study: Response to report of management 
review of TIMSS; Characteristics of national samples and sanq)ling frames; TIMSS objectives 
and their priorities; Report on March National Project Coordinators meeting; International 
dissemination plans* Presentations to the Board on International Comparative Studies in 
Education* February and May. 

1993 lEA Third International Mathematics and Science Study: Overview of sUtus (sUffing at the 
international coordinating center; procedures for scheduling work flow; sanq>ling issues; 
definition of Population 3 for TIMSS; achievement tests; analysis plan). Presentation to the 
Board on International Comparative Studies in Education. June. 

Robitaille, David F., General Editor 

1993 Curriculum Frameworks for Mathematics and Science. By David F. RobiUille, William H. 
Schmidt, SenU Raizen, Curtis McKnight, Edward Britton, and Cynthia Nicol. TIMSS 
Monograph No. 1. Pacific Educational Press: University of British Columbia, Vancouver, 
British Columbia, Canada. 

Robitaille, David, and Gordon Ambach 

1992 lEA Third International Mathenutics and Science Study: Stotus. PresenUtion to the Board on 
International Comparative Studies in Education* October. 

RobiUille, David, Pierre Foy, William Schmidt, David Wiley, Richard Wolfe, Albert Beaton, and Daniel 
Horvitz 

1993 Design Issues For the Third International Mathematics and Science Study. Session chaired by 
Robitaille at the American Association of Educational Research annual meeting* April. 

Schmidt, William H* 

1990 Proposal for U*S. National Center for lEA Third Mathematics Study. PresenUtion tc the 
Board on International Comparative Studies in Education. February. 
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1991 lEA Third IntenuUional Mathematics and Science Study: Report from the U.S. National 
Research Coordinator. Presentation to the Board on International Comparativi^ Studies in 
Education. May. 

1992 lEA Third International Mathematics and Science Study: Status of U.S. participation in 
TIMSS and SMSO; Use of item "signatures* in data analyses; Status of SMSO and U.S. 
TIMSS activities. Presentations to the Board on International ConqMurative Studies in 
Education. May and October. 

1992 TIMSS curriculum analysis: Topic tnwe mapping. Prospects. XXn(3)83. UNESCO. 

1993 lEA Third International Mathematics and Science Study: Report of the U.S. National Steering 
Committee. Presentation to the Board on International Comparative Studies in Education. 
June. 

1993 lEA Third International Mathematics and Science Study: U.S. National Center Update (spring 
item pilot; fall context questionnaires pilot); SMSO activities. Prepared for the Board on 
International Comparative Studies in Education. October. 

1993 The development of a model for the international study of learning opportunities. Presentation 
at the Comparative and International Education Society regional meeting. October. 

1994 International Mathematics and Science Curricula. Presented on the panel on The Third 
International Mathematics and Science Study at the American Association for the Advancemwt 
of Science meeting. February. 

1994 International Mathematics and Science Curricula. Presented on a panel on National and 

International Tests: How Good is the Science at the American Association for the 

Advancement of Science meeting. February. 
1994 lEA Third International Mathematics and Science Study: Curriculum analysis plans. Status 

report and presentation to the Board on International Comparative Studies in Education. May. 
1994 TIMSS Educational Opportunity Model: Detailed Instrumentation And Indices Development. 

University of British Columbia, Vancouver, Canada. 

Schmidt, William, [?] Jorde, and Ted Britton 

1993 TIMSS: A Report on the Development of Teacher Questionnaires finom Five Countries. A 
panel presenting Survey of Mathematics and Science Oppor unity and opportunity-to-leam 
work at the National Association for Research in Science Teaching meeting. April. 

Schmidt, William, Gilbert Valverde, Curtis McKnight, and Ted Britton 

1993 New Trends in the Cross National Con4>arison of Curricula. Panel with papers presented. 
March. Kingston, Jamaica. 

Schmidt, William, SenU Raizen, David Wiley, Eugene Owen, and Larry Suter 

1991 lEA Third International Mathematics and Science Study: Report of TIMSS National Project 
Coordinators meeting. September. 

Selden, Ramsay, Lois Peak, and Eugene Owen 

1992 lEA Third International Mathematics and Science Study: Report of National Project 
Coordinators meeting. Presentation to the Board on International Comparative Studies in 
Education. October. 

Travers, Kenneth 

1994 lEA Third International Mathematics and Science Study: Status report - Population 3. 
PresenUtion to the Board on International Comparative Studies in Education. February. 

Travers, Kenneth, and Jeanne Griffith 

1993 lEA Third International Mathematics and Science Study: Issues of concern to the National 
Science Foundation and the National Center for Education Statistics. Presentation to the Board 
on International Comparative Studies in Education. 
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U*S* TIMSS National Research Center 

1994 International Study of Opportunities to Learn, Including Textbooks and Instructional Practice. 

Special session at the National Association for Research in Science Teaching. March. 
1994 Analysis of Science Curricula from Over Forty Countries: Launching the Third International 

Mathematics and Science Study. Presentation at the NSTA meeting. March. 
1994 Cross-National Measurement of Opportunities to Learn in Math and Science. Presentation at 

the American Educational Research Association meeting. April. 
1994 What in the World Are World Class Standards? Presentation at the American Educational 

Research Association meeting. April. 
1994 International Curriculum Analysis: Results from the TIMSS. Presentation at the American 

Educational Research Association meeting. April. 
1994 Measurement of Opportunity to Learn Elementary and Secondary Science and Mathematics: 

Recent Experience with Methods^ Validity^ and hiterpretation. Presentation at the American 

Educational Research Associaticm meeting. April. 

Valverde, Gilbert 

1993 Development and validation of instruments for assessing opportunities to leam. Presentation at 
the Conqwrative and International Education Society regional meeting. October. 

1994 International Curriculum Analysis: Preliminary Results from the Third International 
Mathematics and Science Study. Presentation at the Comparative and International Education 
Society annual conference. March. 

1994 United States of America: System of education. International Encyclopedia of Education. 
Second edition. 

Wiley, David 

1992 lEA Third International Mathematics and Science Study: Report of the Sampling and 

Methodology Committee. Presentation to the Board on International Con^wrative Studies in 
Education. October. 

Wolfe, Richard 

1992 lEA Third International Mathematics and Science Study: Sampling plans. Presentation to the 
Board on International Comparative Studies in Education. May. 



NOTE: This study summary was reviewed and edited by William Sdimidt at the U.S. National 
Research Center for TIMSS at Michigan State University on July 10, 1994; Schmidt provided 
database information on November 29, 1994. 
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